1 Day Processing
As expected, bureaucracy is alive and well at the US consulate. Since we were told that processing would take 1 day, we thought that flying 9 days later would be safe. Unfortunately, this was not the case.

The original plan had the movers packing for us tomorrow, moving us out on Thursday, and us flying on Friday. However, I've been putting off canceling utilities, paying for the flight, and confirming the movers until the visas arrived. We expected the paperwork yesterday (Monday), and accepted that they might get delayed a day (Today), but still nothing is forthcoming. So once 5pm rolled around we called everyone up and postponed until next week.

Other than the annoyance of waiting longer before we travel, the postponement did reduce some of the pressure initially (especially for Anne). However, it has caused a whole new set of problems. For instance, our mail starts being redirected this Friday, and requires 5 days notice to change (along with a fee). If the visas haven't arrived before the redirection, then they won't get here at all! This is because they are being sent via the Australia Post overnight courier service, which shows up with the mail.

I don't think Express Post items get redirected like normal mail does. Instead, they will be returned to the sender - in this case, the consulate. I think I'd have more luck getting the documents if they followed the redirection to Chicago.

The movers are booked solid next week, so they're coming tomorrow anyway. I'm not sure how we'll get by with everything packed up: plates, cutlery (or flatware), summer clothing, etc. But we'll have to make do.

In the meantime I'm trying to work, but my productivity is steadily declining this week (it wasn't too bad yesterday, wasn't too good today, and I expect it to be non-existent by tomorrow). As such, I don't think I'll be writing much on technical topics for a few days.

Visas
Anne, Nic and I flew to Sydney on Wednesday to visit the US consulate. This was to present our paperwork, and attend a 5 minute interview in order to get our visa. The rules said Anne had to be there, and Nic naturally has to be with Anne. Luc got to stay with his grandparents for the day. Refer to Anne's entry for his day. (That reminds me... I need to tell her to use JPG instead of PNG. Even with low compression the photo is 1/4 the size).

I can only presume everything went well, as they said we would have our paperwork back in a couple of days. This is sure to be delayed an extra day with the Australia Day holiday on the Thursday.

I can't say I enjoyed the process very much. We had to be up at 4:30am in order to get our flight, and we needed to go through security at the consulate early enough to make our "appointment", which was set at 10:30am. What really happened, was that we cleared security with 5 minutes to spare, and then waited in line for several hours before being called up. At that point our documentation was inspected and filed, and we were asked to take a seat again, until the interview.

At this point we've been in there for several hours, having drunk a couple of coffees to help deal with the lack of sleep (Nic is a good sleeper, but he's still a 10 week old baby and needs feeding overnight). We were in a room where all the staff are behind thick glass (why are they so paranoid about security then?) and the only way in or out is a thick security door. The nearest accessible rest rooms were 59 floors away, and on the other side of security. This made for a very unpleasant wait.

Babies being as they are, Nic needed changing after a couple of hours, so Anne took him downstairs and back through security again. Coffee being as it is, I was obliged to follow soon after. I was lucky, since I got back just in time for the interview. Five minutes and $520 later, and it's all over. We grabbed a bite to eat, caught a train back to the airport, waited on the tarmac while they fixed a faulty fuel pump, and came home.

Time from leaving the house to getting home again? 14 hours. What a waste of a day.

Syntactic Sugar
I really like some of the new features in the language for JDK 1.5. Generics, autoboxing, and annotations can make code easier to write and maintain. However, Java has implemented a number of these simply as syntactic sugar, rather than as a real change to the environment. This was important for maintaining compatibility, but sometimes it gets annoying.

While looking for a bug today (it turned out to be a typo that happened to compile) I decided to print out various forms of Unicode. These included a char[], a Character[], and a List<Character>. The code to print these used the new looping construct:

  void print(char[] c) {
    for (char x: c) System.out.print(x);
    System.out.println();
  }

  void print(Character[] c) {
    for (Character x: c) System.out.print(x);
    System.out.println();
  }

  void print(java.util.List c) {
    for (Character x: c) System.out.print(x);
    System.out.println();
  }

Notice how the difference between each method is trivial? If Java used templates in a similar way to C++, then this would be one method. Yes, I know the compiler creates 3 methods in the binary. But why would I care about code redundancy on that scale when Universal binaries make duplicates of every piece of executable code? These days we need space for data, and almost never for code (even embedded applications typically have megabytes available to use).

So generics are nice, but when they can't be used in methods like these, they feel too restrictive.

The other thing that bothers me, is that the only way to describe management of an array is to create static methods that take arrays as parameters, and/or return arrays. Java arrays have no inherent extensibility. Sometimes I find that it would be useful to complement certain classes with another class that holds an array of the first class. Unfortunately, these classes are not compatible with real arrays. At least lists are there to take up some of the slack, but ArrayList can only get you so far.

I'm probably just a little sensitive to the inflexibility of arrays, as I've been (finally) learning some Python, where life is definitely better for arrays. I therefore found it amusing that one of my responses to the sort/unique problem of the other day was from Stephen where he showed of his 1337 h4x0r skillz with Python to solve the problem in 2 lines using built-in language features. Maybe I should be moving to Jython. :-)

Family Life
Anne has recently decided to put up a web page on our domain. She's using iWeb and I've been impressed with how quickly she was able to put together a site that looks reasonably attractive. I just hope it doesn't look like the web site of everyone else who uses iWeb. :-)

This will only be of interest to people who know us personally, as it is very focused on our family. I have no control over content, though I have been encouraged to read it myself. I think Anne just wants someone to proof read. ;-)

Unicode
In my last post I was converting arrays of char into arrays of java.lang.Character. However, under Java 1.5 this runs into a new problem.

In Java 1.4, the Character class was based on Unicode 3.0, which defined Unicode characters in a 16 bit space. In Java 1.5 this has been revised to Unicode 4.0, which defines Unicode in a 21 bit space. Unfortunately, the 1.4 implementation of Character wrapped the 16 bit char native type, and is not extensible enough to deal with Unicode 4.0.

Instead of creating a new class to deal with this, Character has included a new set of static methods to work with the larger characters. The result is a little messy.

Character now works with 32 bit integers, which it refers to as CodePoints. Most of the static methods which take a char have been duplicated to accept an int as a CodePoint. This all works from the perspective of the static methods, but there is no longer a class that represent a complete Unicode character. The closest available is java.lang.Integer, which is fine for storing the value, but has no support for character operations.

I first realized this problem when I tried to convert from a String.toCharArray() into an array of Character. The new Character class has a static method for reading from a Unicode character from a char[], which holds characters in UTF-16 format. This made me wonder if the char[] from String.toCharArray() were actually being returned in UTF-16. This isn't explained explicitly, but the documentation for String indicates that the result is in UTF-16.

UTF-16 refers to an array of char where most Unicode characters (or CodePoints) fit into a single char or Character. However, if a character falls into a particular range, then it is not a complete Unicode character on its own, but instead it is merged with the following char to form a full 21-bit value. A pair of characters like this is called a surrogate pair.

Since an individual char is no longer guaranteed to represent a complete CodePoint, it is not possible to sort an array of char or Character as previously described. Instead, each character has to be decoded into a CodePoint first, and the array of CodePoints can then be sorted. Only there is no such object as a CodePoint, so an array of Integer has to be used instead.

But what about the method to convert a char[] to a Character[]? This now needs to be updated to convert a UTF-16 char[] into and array of objects which are large enough to handle full CodePoints. There also needs to be a corresponding method to convert back to a UTF-16 char[]. With all this work, why not create a new class for CodePoint, which includes these methods, plus all the methods appropriate to manipulating CodePoints?

Again, all the iterative code can be encapsulated in methods to hide the details, but now these methods can be attached to the object in question. This time the code is more complex, as it needs to test each character to see if it is part of a surrogate.

I've coded most of the CodePoint class, but I still have to bring over a few of the static methods from Character and make them instance methods on CodePoint. There are a LOT of methods, so I don't see myself writing tests for all of them. Maybe I'll have a go if other people are interested.

This may be overkill, given that only a few languages need characters which fall into the surrogate range. All Latin based languages are completely covered by char and Character. However, all it would take would be a single extended character to break an application which didn't handle all CodePoints. We've had comments about appropriate Unicode support for Chinese characters in Kowari before, so it's a legitimate concern.

Loops Considered Harmful
Yesterday someone from work told me that he was trying to take all the characters in a java.lang.String, and return a string containing a single instance of each of those characters, in sorted order.

He had some straightforward code to do the work, but he had used a boolean array of ASCII bytes, and couldn't handle UniCode chars. In my cleverness, I wanted to show him how to do the work in 2 lines of code, only I immediately started running into limitations within Java.

Why two lines? Well it should be possible to put the contents of a string into an object that can sort them and remove duplicates, and then pull this data back out and put it into a new string. Learning Lisp I've learnt to expect the sort of functionality needed to do this simply and easily. There shouldn't be a need to iterate over the characters, recording the ones in use, and manually sorting. The data being manipulated should be a single "entity", rather than arrays of elements that require iterative techniques to manipulate. This functionality should all be available, making the final function a trivial call to just a few methods. As Abelson and Sussman said, "Programs must be written for people to read, and only incidentally for machines to execute." Iteration, sorting, and other mechanical operations should all be services found in lower levels of the code, and abstracted away.

Incidentally, this is exactly the sort of thing that is done all the time with the STL in C++. Because all of the collection classes (and arrays) meet the required interfaces, then almost any data type can be used in any position. All you need here are algorithms for sorting and removing duplicates, feeding in a string and getting one out the other side. Admittedly, C++ contains language features which allow significant abuse (and I've seen it abused quite badly), but it also allows for some very elegant code. Since C++/STL is an "unsafe" language offering some of the functionality you expect to see in Lisp, then this brings to mind the quote from Philip Greenspun: Any sufficiently complicated C or Fortran program contains an ad-hoc, informally-specified bug-ridden slow implementation of half of Common Lisp.

Back to Java, the obvious way to handle a problem like this is with a java.util.SortedSet (the only implementation of this in java.util.* is TreeSet). I just had to get the characters out of the string, insert them into the SortedSet and get them back out again, in String form. You'd think this would be easy, but I encountered several problems, showing up problems in the Java libraries, and even the Java language.

The String, the characters retrieved from it, and the sorted set of characters should all form "entities", which are simply different representations of exactly the same data. Since Java and its libraries expect each of these representations at different times, it should be easy and trivial to go from one representation to another. However, this isn't the case.

Strings can provide their contents as an array of char. However, these SortedSets need to take objects as their basic elements. (I understand why the language is built this way, but it still frustrates me that I can't declare a SortedSet<char>.) The object that wraps a char is java.lang.Character, so the array of chars will need to be converted to Characters before they could be put into a SortedSet. Also, since we want to treat the data as a single entity, we want to avoid iterating through the data, and instead insert the entire entity into the SortedSet. This means converting a char[] into a Character[].

Autoboxing
While Java 5 now supports autoboxing between native types and their associated objects, there is no way to convert an array of a native type into an array of associated objects. Since arrays are a part of the language, and not an accessible class, there is no way to extend arrays into doing this either. Instead, a loop must be written to iterate through one array, converting the elements to the types in the other array. This is an easy thing to do, but it's the kind of fundamental operation that should be found in a library, rather than being re-implemented by every developer.

So the first thing I had to do was write a method to autobox a char[] into a Character[] and another method to unbox it back again. These are quite trivial:

  private static final Character[] toCharacters(char[] chars) {
    Character[] characters = new Character[chars.length];
    for (int i = 0; i < chars.length; i++) characters[i] = chars[i];
    return characters;
  }

  private static final char[] toChars(Character[] characters) {
    char[] chars = new char[characters.length];
    for (int i = 0; i < characters.length; i++) chars[i] = characters[i];
    return chars;
  }

So this will create an entity that holds all of the characters in java.lang.Character objects, rather than the char values that come from a String. Unfortunately, Java arrays are not compatible with the collections classes (unlike other languages, like Python, or C++ when using the STL), so the array needs further conversion still. This time there is a utility to convert an array into a collection-compatible object, called java.util.Arrays.asList(). So now I can finally put the string into a SortedSet:

SortedSet charSet = new TreeSet<Character>(
      Arrays.asList(toCharacters(inputString.toCharArray())));

The java.util.Collection interface also specifies the toArray() method for converting the collection into an array, so it is possible to move back this way as well. However, the String class still needs to work with char rather than java.lang.Character, so the array unboxing method above is needed to convert this way:

public static String sortedWithoutDuplicates(String inputString) {
    SortedSet<Character> charSet =
        new TreeSet<Character>(
            Arrays.asList(toCharacters(inputString.toCharArray())));
    return new String(
        toChars(charSet.toArray(new Character[charSet.size()])));
  }

It's still disappointing how ugly this looks at the end. The first line has two more method calls than it should need:

1. toCharacters() to convert the char[] from inputString.toCharArray() to a Character[].
2. Arrays.asList() to convert the Character[] to a List<Character>

Instead, it would be nice to see constructors for collections accepting arrays. Since the language does not support array boxing/unboxing, then It would also be good to see a method like String.toCharacterArray(). With just those two changes the first line would instead be:

  SortedSet<Character> charSet =
      new TreeSet<Character>(inputString.toCharacterArray());

Similarly, the second line has some ugliness to it as well:

1. java.util.Collection<E> defines the methods:
   • <T> T[] toArray(T[] a)
   • Object toArray()
   We have to use the first method if we want to get back a Character[], but this means we need to create the array manually before calling the method.
2. toChars() to convert the Character[] into a char[] for the String constructor.

It would be nicer if the Object toArray() method had be redefined into: E[] toArray(). I can't see why this change wasn't made with the release of generics, since E[] is still a valid Object[], and the other method got redefined this way when generics came out. I'd also like to see String accept a Character[] in a constructor, in the same way I'm looking for a String.toCharacterArray() method. These changes would make the second line look like:

  return new String(charSet.toArray());

So my fictional extensions would turn the three methods above (toCharacters(), toChars() and the mess in sortedWithoutDuplicates()) into the following single method:

public static String sortedWithoutDuplicates(String inputString) {
  SortedSet<Character> charSet =
      new TreeSet<Character>(inputString.toCharacterArray());
  return new String(charSet.toArray());
}

As a postscript, I should explicitly point out that I don't think that the constructor for TreeSet<Character> should take a String, nor should a String constructor accept a SortedSet<Character>. These operations are too specific to the described task, and do not merit inclusion in a standard library. In particular, they would only apply if the wrapped type of the SortedSet is Character, and would break down on other types.

Why would I make suggestions about modifications to the standard libraries? While we have to work with them, we shouldn't just lie down and accept the shortcomings of Java. It doesn't happen all the time, but Sun do listen to suggestions on the language and libraries, particularly when suggestions extend the standard libraries without breaking them. Also, if they ever release Java as open source software, then we will have an opportunity to make contributions which may even get accepted.

Universal Binaries
Now that Macs are finally available with Intel chips, it occurred to me to give serious consideration to Universal Binaries. I never considered it a big deal, since XCode handles that for you. But all my recent binary code (non-Java) has been built with Makefiles so it can be compiled for multiple platforms. So I thought I'd better learn how to do it from the command line.

My first thought was that gcc might have some sort of "Mac-universal" option for the -arch flag. The man page says:

-arch arch
       Compile for the specified target architecture arch.  The allowable
       values are i386, ppc and ppc64.  Multiple options work, and direct
       the compiler to produce ``universal'' binaries including object
       code for each architecture specified with -arch.  This option only
       works if assembler and libraries are available for each architecture
       specified.  (APPLE ONLY)

I naively thought that -arch i386 -arch ppc would work. However, the result was just a PPC file. The man page for ld had the answer for this:

       The link editor accepts ``universal''  (multiple-architecture)  input
       files,  but  always  creates a ``thin'' (single-architecture), standard
       Mach-O output file.
    ...
       Only one -arch arch_type can be specified.

So how were universal binaries made if the linker can't do it? Again, the man page for ld provided the answer:

       The compiler driver cc(1) handles  creating  universal  executables  by
       calling  ld(1)  multiple  times and using lipo(1) to create a ``univer-
       sal'' file from the results of the ld(1) executions.

So it looks like the build process starts in the same way that it always did, only now it repeats the process for the other architecture, and merges the final executables. This means that intermediate files either need to be in separate directories, or else they need to have different names (which doesn't sound like a good idea to me). It also includes the new lipo tool for merging. (I can't think of what this might be short for. I keep thinking "liposuction".)

I tried using this technique to build a universal binary for a "Hello World" program, but kept getting missing symbols for the linker stage on the i386 architecture. A look at the compiler log for an XCode project showed an option of -isysroot /Developer/SDKs/MacOSX10.4u.sdk. This isn't mentioned in the manual for gcc or ld, but it does the trick.

To illustrate the process, a simple program might be built from two files: first.c and second.c

  gcc src/first.c -c -o obj/first.o
  gcc src/second.c -c -o obj/second.o
  gcc obj/first.o obj/second.o -o bin/program

(I've used gcc to handle linker arguments automatically for me.)

The same build for a universal binary would look like:

  gcc src/first.c -c -arch ppc -o obj/ppc/first.o
  gcc src/second.c -c -arch ppc -o obj/ppc/second.o
  gcc obj/ppc/first.o obj/ppc/second.o -o bin/ppc/program
  gcc src/first.c -c -arch i386 -o obj/i386/first.o
  gcc src/second.c -c -arch i386 -o obj/i386/second.o
  gcc obj/i386/first.o obj/i386/second.o -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -o bin/i386/program
  lipo -create bin/ppc/program bin/i386/program -output bin/program

lipo can also be used with the -info flag to check a file type:

  $ lipo -info bin/ppc/program  
  Non-fat file: bin/ppc/program is architecture: ppc
  $ lipo -info bin/i386/program
  Non-fat file: bin/i386/program is architecture: i386
  $ lipo -info bin/program 
  Architectures in the fat file: bin/program are: ppc i386

This is all fine for Apples, but how should I write a portable Makefile that does all of this? Looking at a current Makefile, I really want to define the architecture in variables, and call into the same Makefile twice, followed by the lipo command to merge the results. However, Makefiles are not as easy to program as most languages. The biggest problem for me is how to call into another Makefile regardless of which target you are currently making. A test for "Mac OS X" at the top of the Makefile will not work, as commands can't be placed outside of a "target" block.

The only way I know to do this is to test for "Mac OS X" in every target, and duplicate the compilations. This would be tedious to write, be prone to errors, and be awkward to update (particularly for non-Mac users). I suppose I need to learn a lot more about Makefiles, but I'd rather not have to.

This could be easier in Ant. After all, the code I'm writing is actually a JNI library. Maybe I should write a gcc module?

CAS
The interfaces in UIMA have seemed rather obtuse recently.

To process a document, an analysis module is given a CAS (Common Analysis System), and returns a CAS. The CAS contains a reference to the original document, and any annotations that have been made so far. A module can ask the CAS for the original document, perform its analysis, and add any annotations back into the CAS. At the end of the process, the UIMA framework returns a CAS object which can be checked for annotation properties.

Adding annotations to a CAS is an easy process for a module. An appropriate Annotation derivative is created for the current CAS (the CAS object is passed as a parameter to the Annotation's constructor). It is also easy to read these annotations after UIMA has finished. However, the CAS objects in each case are accessed through different classes, with different features. In an analysis module I have a JCas object, but the results of the UIMA run are returning a TCAS object instead. TCAS is actually just a specialization for handling text of a standard CAS object, but I didn't see that immediately (I should have paid closer attention).

Fortunately, a JCas object has a method called getCAS(), and a CAS has a getJCas() method, as these objects have a 1:1 correspondence. Why have two objects? I think it's to separate out the CAS concepts from the Java specific information needed to manipulate a CAS. But don't quote me.

Separating these two classes, and providing different types in different circumstances has tripped me up a bit. Maybe someone can explain the reasoning, but I've just found it annoying.

Speculation
The blogosphere has started picking up some of the hassles between Kowari and NGC, so I figured that it's my turn to comment.

Last year, NGC asked for a 1.2 release to be pushed back to the 9th of January. Being an Open Source project, NGC could not force this postponement, even though they now own the copyright for the core of the code, and have initial contributor status (as defined in the MPL). But Open Source is new for NGC, and we wanted to support them supporting us, so we accepted the delay.

Then last week an NGC lawyer asked David to push it back further, citing "irreparable harm" to their company if the release proceeded as scheduled. To me, this speaks of a misunderstanding of the entire Open Source development process. The code and documentation is already out there for anyone to see. Just do a cvs update on Sourceforge. A "release" in this context simply means a marker on the code to indicate a certain level of features and stability. (We did a little more as developers, due to the code freeze, testing, and so on, but that doesn't influence users directly).

With NGC giving inappropriate directives to David, and given the work he has done with them, he decided that his best course of action is to leave the Kowari project.

While I don't think that NGC are acting to squash the Kowari project, they have put some serious blocks in the path of administering it. This has stemmed from them not understanding Open Source development, nor the MPL licence that Kowari has been released under. This is understandable, given their corporate/defence background, but it is still frustrating.

Since NGC are not aware of how an Open Source project works, it is conceivable that they would try to interfere with users' ability to continue to use and extend Kowari. Fortunately, the MPL offers protection for everyone here. Any challenges to a user's rights under this licence would necessitate a number of large corporations coming to the defence of the MPL, among them would be AOL and IBM.

Similarly, the MPL protects developers. We still have copyright on personal contributions (which excludes those contributions made while working for the Initial Developer). Those bits can be used elsewhere. There are also rights to do other things like "forking" the project. Personally, I'd rather not do anything like that, as it means abandoning the reputation that Kowari has built up over time, and antagonising NGC in the process.

The situation is annoying for developers (and untenable for David), but it shouldn't mean big problems for users. Kowari, or something like it, should continue. In the worst case, a snapshot of the currently accessible source (January 10, 2006) will always be usable by the community.

I haven't done any Kowari work in the last month, and probably won't do much until I get to Chicago next month. After that, I plan on continuing developing for Kowari, or a project like it.

Domains
I've always thought that domain squatting was an unethical way to make money, but had only heard stories of it before now.

Anne suggested that it might be nice to pick up the gearon.com domain, if it was available. After all, I normally take up a lot of the first result page at Google when you type in my surname. (Just looked, and today I don't! I really need to blog more often.) OK, so I'm not a commercial entity, but everyone recognizes .com, while people often look at me funny when I say .org or .net. The .com thing has brand recognition.

So I had a look, and discovered that the domain is already registered, but it is for sale. It turns out that it's available for purchase through the bidding process available at Afternic.com. The minimum bidding price was way more than I'd have liked to spend, but it's my name, so why not?

A week later I discovered the bid was rejected. I asked Afternic what a decent price is supposed to be (according to their market analysis). Their response was that the current market value is $200. Still far too much, but it gave me confidence to ask the current domain holder how much they would like.

The answer? $2950. Quoting from the email:

Price is very low for a family name.

Huh? Whose family? The Rockefellers?

I didn't care about the domain all that much (it should probably go to a more commercial interest, like something run by Michael Gearon or Tierney Gearon), but registering a name and then charging to give it back to an owner of that name is a principle I find rather offensive. I suppose I should be grateful she was asking for $3000 and not $30,000.

I resolved it by registering gearon.org for $8.20.

UIMA
The other day I followed a link over to IBM's DeveloperWorks, and found that they have an RSS feed for their tutorials. I was pleased to find a simple Python tutorial that I'm using to finally introduce myself to that language. But more importantly, I found a tutorial for generating a UIMA annotator.

The UIMA docs are very verbose, and a tutorial like this has been great for cutting through the chaff. It's still full of stuff I don't need (mostly because I've already learnt it from the official UIMA docs), but it's still been a real help.

My biggest problem at the moment is that UIMA wants all my annotations in character offsets. Unfortunately the library I'm using is providing my information in word offsets. That's trivial to convert when words are separated by whitespace, but punctuation leads to all sorts of unexpected things, particularly since the grammar parser treats some punctuation as individual words, while others get merged into existing words.

I'm starting to wonder if I need to re-implement the parser so I know what the character offsets of each word will be. Either that, or I'll be doing lots of inefficient string searching. I don't find either prospect enticing. Maybe if I sleep on it I'll come up with something else.

Gödels Theorem
I was just looking at the fascinating exhibit of equations by Justin Mullins. I'm not sure if I see the exhibit as art, since the visual appearance evokes little in people who do not understand the equations (with the possible exception of the Four Color Theorem), but they are certainly beautiful.

I particularly loved the end of the narrative for Gödel's theorem:

Others have wondered what Gödel’s theorem means for our understanding of the human mind. If our brains are machines that work in a consistent way, then Gödel’s theorem applies. Does that mean that it is possible to think of ideas that are true but be unable to prove them? Nobody knows.

Note the sentence that I highlighted. If it is true, then that sentence is an unprovable idea. I love it.

Qubytes
Lots of places are commenting on the new quantum memory chips in silicon.

I'm surprised at this. I expected that embedding quantum devices in silicon would be done with quantum dots, rather than ion traps. It is probably better than it was done with ion traps as there seems to have been more research into quantum processes using this technology. After all, what good is a quantum state if you can't apply transformations on it without collapsing the state?

All the same, a chip like this is just a first step in a long line of problems to be solved. There is no discussion about setting up quantum states, nor reading them back. There is no discussion about the ability to entangle the qubits on the chip, and how far that will scale. Transformations will eventually have to be built in to the chip. But if research has taught me anything, it's that the big problems are usually solved by lots of people chipping away at the little problems. By the time the final solution comes around, it doesn't seem like a big deal any more.

Tracker
The little tracker icon I have on this page is a link to a service that tells me how many hits the blog is getting (but not the RSS feed). I haven't bothered to look at the stats in a long time. After all, I'm rarely writing, so why would anybody (beyond my friends) bother to read?

Apparently I was wrong in that assessment. I'm averaging over 20 hits a day, with peaks over 40, and I'm writing less that once a week. My infrequency is due to lack of time. Given how many people are reading so little here, I'm wondering if anyone else suffers the same problem. :-)

Blogging
I notice a new post on the Google Blog describing a new tool for Firefox. When installed, a small message will appear on any page you visit, showing a list of blogs which refer to that page. Sounds cute.

I normally browse with Safari (gotta love those native widgets), but I keep Firefox installed (after all, some pages have extra features when viewed with Firefox). So here was a chance to upgrade my version of Firefox (I hadn't picked up 1.5 yet) and install Google's new tool.

So where should I go first to check out the comment? Well obviously my usual home page of Google comes up, and there are ample comments. How about the page talking about the new tool? Lots of comments there too. Oh, I know! How about this blog? :-)

Unfortunately the list of comments was a little disappointing, mostly including my friends. That will teach me to not blog regularly. However, I did find one blog on semantic web development that I found really interesting. I was just disappointed that he didn't have a lot of incoming links (though there was one worth checking out).

All in all, it's a tool that I like. In fact, I wouldn't mind a similar tool that did a link: search on Google, rather than just in the bloggosphere.

Modeling Talk
Last week I was invited along to a talk given by Bob at SAP. I enjoy seeing what Bob's working on when I'm not discussing OWL with him. He's a clever guy, and understands modeling quite well. I just wish I'd written about it sooner, as I won't be so clear anymore.

Probably the most important thing I got out of his talk was an overview of category theory. Andrae and Simon have both spoken about it, and I've come to understand that it's relevant, but as yet I haven't learnt anything about it. Bob gave the 30 second overview for computer scientists, which I found quite enlightening.

I finally got my copy of Types and Programming Languages (otherwise known as TAPL), and have been looking forward to reading it. But when ordering this book I discovered that Benjamin Pierce has also written a much smaller book called Basic Category Theory for Computer Scientists. I had considered getting this book (at only 117 pages, it looks like a relatively quick read), and Bob's talk has now convinced me. The only problem is that I'll have to put the order off until we move to the States... whenever that happens.

modeling
Speaking of books, I also picked up a copy of MDA Distilled, Principles of Model-Driven Architecture. Some of the work I did with SAP came out of this book (virtually guaranteed when you work with one of the authors), and it talks about the kind of dynamic modeling that I've been talking about investigating with OWL. I haven't been through all of it before now, so I thought it would be worthwhile reading it in detail.

Coincidentally, I was explaining some of my ideas to someone at work today (Indy), and referred to this book to describe some of the background. I had some idea that Herzum Software worked with MDA (which is why I thought they might be interested in this work), but I had never thought of it as a formal association. Indy quickly made it very clear that Herzum Software specifically put themselves out there as an MDA company. That makes perfect sense as it aligns with what I already knew, but being in my own little corner of the world has kept me isolated from the advertising of it. Anyway, it's nice to know that the direction I'm moving in is paralleled by the work of my new employer.

RDFS Entailment
I've also been in an email discussion about entailment on RDFS. It seems that the following statements:

  <camera:min> <rdfs:range> <xsd:float>
  <_node301> <camera:min> '15.0'^^<xsd:float>

will lead to an entailment of:

  <xsd:float> <rdfs:subClassOf> <rdfs:Resource>
  '15.0'^^<xsd:float> <rdf:type> <xsd:float>
  '15.0'^^<xsd:float> <rdf:type> <rdfs:Resource>

It seems that I didn't cover all the possible rules which could lead to a literal in the subject position. It's quite annoying, as these are completely valid entailments, according to RDF semantics. Making special cases to avoid particular results seems like a hack.

In a similar way, it seems wrong to not allow entailments about blank nodes. I should re-visit the decision there. I think I need to re-read the semantics document to see if I can get further enlightenment. At the least, I know that I can't entail a statement with a blank node as the predicate. Like the problem with literals, the semantics document appears to justify this sort of statement, but the RDF syntax doesn't allow for it. I know this is a particular bugbear for Andrae.

Catch Up
I've been wanting to write for nearly a week now. Every time I try to sit down for it I've had a work task, family needs, or packing to take priority over this blog. I ended up having to write little notes to myself to remind me of what I wanted to blog about.

JNI and Linux
Having made the Link library work on Mac OSX using JNI, I figured it would be easy to get it working on Linux as well. Unfortunately it didn't work out that way.

To start with, I got an error from the JVM saying that it could not find a symbol called "main" when loading the library. This sounded a little like dlopen loading an incorrectly linked file. I'm guessing that the dlopen procedure found what it thought was an executable, and therefore expected to see a main method. Googling confirmed this, but didn't really help me work out the appropriate flags for linking to fix this.

I had compiled the modules for the library using -fPIC (position independent code). I then used a -Wl,-shared flag to tell gcc to pass a -shared flag to the linker, in order to link the modules into a shared library. However, it turned out that I really needed to just use -shared directly on gcc. I've still to work out what the exact difference is, but that's not a big priority for me at the moment, since I have it working. According to DavidM there is something in the gcc man page about this, so at least I know where to look.

After linking correctly, the test code promptly gave a Hotspot error, due to a sigsegv. This meant that there was a problem with the C code. This had me a little confused, as it had run perfectly on OSX. Compiling everything in C and putting it all in a single executable demonstrated that the code worked fine on Linux, so I started suspecting that the problem might be across the JNI interface. This ended up being wrong. :-)

There are not many differences between the two systems, with the exception of the endianess of the CPUs. However, after looking at the problem carefully, I could not see this being the problem.

The initial error included the following stack trace:

C [libc.so.6+0xb1960]
C [libc.so.6+0xb4fcb] regexec+0x5b
C [libc.so.6+0xd0a98] advance+0x48
C [liblink.so+0x19f9d] read_dictionary+0x29
C [liblink.so+0x1d705]
C [liblink.so+0x1d914] dictionary_create+0x19
C [liblink.so+0x286c9] Java_com_link_Dictionary_create+0xc1

The only code I had real control of was in Java_com_link_Dictionary_create, dictionary_create and read_dictionary. I started by looking in Java_com_link_Dictionary_create and printing the arguments, but everything looked fine. So then I went to the other end and looked in read_dictionary.

I was a little curious about how read_dictionary was calling advance, as I hadn't heard of this function before. Then I discovered that the function being called was from the Link library, and has a signature of advance(Dictionary). This didn't really make sense, as my reading of the stack trace above said that advance came from libc and not the Link library (liblink). This should have told me exactly what was happening, but instead I tried to justify what I was seeing. I convinced myself that the function name at the end of each line described the function that had called into that stack frame. In hindsight, it was a silly bit of reasoning. I was probably just tired.

So to track the problem down I start putting printf() statements through the code. The first thing that happened was that the hotspot errors changed, making the error appear a little later during execution. So that meant I had a stack smash. Obviously, one of the printf() invocations was leaving a parameter on the stack that helped the above stack trace avoid the sigsegv. OK, so now I'm getting some more info on the problem.

It all came together when I discovered that I was seeing output from just before read_dictionary() called advance(), and from just after it, but not from any of the code inside the advance() function. At that point I realised that the above stack trace didn't need a strange interpretation, and that the advance() that I was calling was coming from libc and not the local library.

Unfortunately, doing a "man advance" on my Linux system showed up nothing. Was I wrong about this method? I decided to go straight to the source, and did a "nm -D /lib/libc.so.6 | grep advance". Sure enough, I found the following:

  000b9220 W advance

So what was this function? Obviously something internal to libc. I could download the source, but that wasn't going to make a difference to the problem or the solution. I just had to avoid calling it.

My first approach was to change the function inside Link to advance_dict(). This worked perfectly, and showed that I'd found the problem. However, when the modules were all linked into a single executable it had all worked correctly, and had picked up the local function, rather than the one found in libc. Why not?

I decided that if I gave the compiler a hint that the method was local, then maybe that would be picked up by the linker. So rather than renaming the function to advance_dict(), I changed its signature from:

  int advance(Dictionary dict)

to:

  static int advance(Dictionary dict)

I didn't know that this would work, but it seemed reasonable, and certainly cleaner since it's always a bad idea to presume that your name is unique (as demonstrated already). Fortunately, this solution worked just fine.

DavidM explained to me that static makes a symbol local to a compilation unit (which I knew) and was effectively a separate namespace (which I also knew). He also explained that this "namespace" has the highest priority... which I didn't know, but had suspected. So I learned something new. David and I also learnt that libc on Linux has an undocumented symbol in it called advance. This is worth noting, given how common a name that is. As shown here, it is likely to cause problems on any shared library that might want to use that name.

There's more to write, but it's late, so I'll leave it for the morning.

Blogging
I'm a little annoyed at myself for lack of blogging recently. This is particularly the case as I see mainstream media commenting on people's online presence more and more. I almost feel like I'm missing out on something. Yes, I know that's a ridiculous concern, but I'm allowed to worry about anything I want to. :-)

Other than the restrictions imposed on me by my recently expanded family, my main problem with blogging recently has been lack of material. I don't mean that I have nothing to say. Instead, I'm limited by what is appropriate to put into a public forum. It was much easier when I worked on Open Source software all the time.

For instance, this last week has had me reviewing software produced by a group of academics at another company. My review is for my employer, so I obviously can't publish it (otherwise, why would he be paying me?). Also, any review will naturally say both good and bad things. The good may be OK, but saying something bad in public is obviously inappropriate. After all, these guys are out to impress customers and make money too.

So I'm left having to write about what I do out of hours. That's all well and good, but having a young family reduces the time for that.

I could always write a few opinion pieces. Australian federal politics has had me feeling frustrated for some time now, and I definitely have things to say on the topic. But that's not what this particular blog is about. I could always start a parallel blog, but then, who would really want to know what I think about Brendan Nelson and higher education in Australia? It would be cathartic for me, but not so much that I think it's really worthwhile.

All the same, I might consider a second blog to contain random musings (like this one). Maybe one evening when I'm not feeling like going to bed, and I have something I feel I want to say. I could be a mix of my daily life, frustrations, and comments on the oft explored experience of fatherhood. I'm not sure it will be good reading, but I may have fun coming back to it in a few years to see just how naive I really was back in 2005. :-)

Grammar
Meanwhile, I'm back to grammar parsing, using Link. I was a little chuffed to get the JNI all working, particularly when I was able to rewrite some of the test code in Java and have it all run correctly. I still need to test that it runs fine on Linux, but I don't have any real concerns there. Making it run on Windows will be another story.

Ideally, I'll be able to use MingW as the compiler, as it should help keep the codebase and build process consistent. I just hope I won't have to jump through too many hoops to generate a DLL file.

I could always ask someone at work if we have an MS commercial compiler, but we may not. I have my own, but I'm notlicensedd to use it for work. It amazes me that people are concerned about the restrictions of Open Source licensing, when commercial licensing can be far worse.

Weather
I'm a little obsessed with the weather at the moment. I enjoy our sub-tropical climate here, and it's going to be a rude shock to land in Chicago in the middle of Winter. As a result, I'm enjoying every minute here that I can. I'm also comparing the weather between the two cities on a day-by-day basis. The huge difference fascinates me, but the guys at work are probably annoyed with me by now.

According to AccuWeather.com, Chicago is currently well below zero Celsius), and will be staying that way all week. The town I grew up in (Chinchilla) often goes below zero during Winter, but that only happens overnight. Chinchilla also hasn't had snow since the early 1900's (1915 rings a bell for some reason).
Brisbane has been my home for the last 17 years, and it has never been below freezing (at least, not in recorded history). So I really haven't experienced anything like Chicago before. Can you blame me for paying attention to the differences?

In the meantime, Brisbane is just starting on its first heat wave for the Summer. Fortunately, it's not supposed to get as high as 40C (104F) over the coming week, but it won't be far off. The prediction is 37C (99F). Not too bad, but unpleasant all the same. Overnight minimums are over 20C (68F), so Luc isn't sleeping too well. This is a far cry from Chicago, where the highest maximum for the coming week is -4C (24F).

This will certainly add some excitement to the move!

Bytecodes
DavidM helped me to find a slew of bytecode libraries (many of which are here). Some of these are better than others, but I was surprised to discover that none of them work from the basis of an AST. That means a lot of work gluing an AST onto an appropriate bytecode library, which reduces the advantages of using third party libraries.

That leads me back to looking at an existing compiler, as these must already go from AST to bytecode. All I'm trying to achieve beyond this is the ability to persist and retrieve the AST in RDF, and a public API for modifying the AST. So maybe I should be going directly to a compiler like the one built into Eclipse?

Types and Computer Languages
One of DavidM's first suggestions to me was to use the expression library from Kawa. While it doesn't express an AST either, it does meet much of the criteria of what I'm looking for. However, it is really centered around providing support for Scheme in a JVM.

I don't really know Scheme, and my first thought was that this would be something else I'd rather avoid learning (there's so much to learn these days - I have to be discriminatory). However, Andrae was quick to point out that it's a type of Lisp, and that I already have the basics in the lectures on "Structure and Interpretation of Computer Programs" (which I have the video files for). He also pointed out that given the Lisp heritage of OWL that I'd do well to learn Scheme. So I decided to pull the lectures back out, dust them off, and watch them right through (I'd only seen the first 4 before). It's not always easy to find the time, but I have to say that I enjoy watching them.

Fortunately, I have a new iPod (yes, that involved some wrangling with Anne), so I've converted all the lectures over to it (courtesy of FFmpeg) and can watch it whenever I have a spare moment. It's just a shame that the battery can only handle a couple of hours of video.

While discussing languages with Andrae, he mentioned Types and Computer Languages by Benjamin Pierce. I've always been impressed with Andrae's knowledge of the theoretical underpinnings of languages, and had put it down to extensive reading on the topic. This book is apparently one of the central sources for the information, so I'm thinking I'd like to read it.

I went looking around the appropriate bookstores in Brisbane yesterday, and half of them did not even have access to a distributor for this book. When I finally found one, they told me it would take 6 weeks to come in from America, and would cost $185 AUD. Looking at Amazon, I'm able to purchase the book and its sequel for only about $165 AUD (and that's probably a similar shipping time). So I'd be buying the book from there, if it weren't for the fact that we're about to move! Would I ship the book here, to my in-laws near Melbourne (where I'll be working for a couple of weeks around Christmas), or to the office in Chicago?

The visa paperwork is very frustrating. It would be nice to know when I'm moving for real. <sigh>

In the meantime I'm reading Java Puzzlers in my Copious Free Time, and enjoying it thoroughly. It's the puzzles that I couldn't answer on my own that I enjoy the most. I've been bugging all my friends with them. :-) The best part is that it's finally encouraged me to read the JVM specification.

Knowledge Mining
Today I dropped into UQ for a short seminar by Osmar Zaiane, which he presented on Knowledge Mining. I'm glad I went, as it discussed several techniques of semantic extraction. In particular, it described the details of several methods based on Apriori.

It also served to remind me about how much I really remember about neural networks (more than I care to admit), and that they're still considered the best solution to some classification problems. Perhaps I should revisit them.

Some of these techniques, plus a few others that were mentioned, may help me to boost the level of semantic extraction I've been able to get so far at Herzum. I'll have to look into this area a little more.

While at the university I dropped in a form to defer my enrolment for a couple of months. With the new job, the new baby, Christmas, and the impending move to Chicago, I figured I might be able to use a short break. Bob agreed that it sounds like a good idea. So I'm officially off until April, though I expect to be working on Kowari and OWL a little in the meantime.

Projects
I've spent my last week and a bit on JNI. It was fun when I started (since I haven't done much C lately), but ultimately tedious. After all, I'm just writing a wrapper around someone else's library. Sure, there are some cute challenges, but it's not quantum physics (that reminds me... I really need to get back to postgrad physics at some point. I should finish this Masters).

In the meantime, I've been looking at a project for web services to keep my mind fresh. Or more precisely, I've been looking at another project when I'm not working or looking after the boys (yes, I'm averaging below 5 hours sleep per night). It's a project I've mentioned on and off for a while now, but I've decided to do something about it.

The principles of the project came about through a combination of several factors. I've had some experience with an application server (where I wrote the JMX code). I've also been working with OWL, from a theoretical point of view, an inferencing perspective, and using it for modeling in several systems. I've been working with UML modeling, and writing an OCL interpreter to manage those models at runtime. I'm also an engineer who really likes bit-banging as opposed to all this high-level abstraction stuff (though the high-level stuff does have its charms). Finally, I've had several interesting conversations with people who've helped me crystallize what I'm trying to achieve.

The Idea
The idea is to use OWL to describe a class that can be instantiated in Java. There are several aspects of OWL that don't make it ideal for this kind of modeling, but it is still possible to use it. This is demonstrated by the ability to map most UML features into OWL. Fortunately, the flexibility of RDF allows almost any conceivable type of annotation to be added to the class, filling in any areas where OWL is not up to the task.

What would be the point of this? Well the first things that comes to mind are Web Services (thanks to David for this suggestion). Currently, services can described in OWL-S. If a client does not know about a described service, then it can always try to model it (this technology is still in development). However, why simulate the model, when you can instantiate a class that meets the description of the model? This would perform better, and offer much more flexibility to the client system.

But How?
One approach for this would be to convert the OWL class into some implementing Java source code, write it to disk, and convert it into a class file with javac. I've never liked this approach. It is very slow, relies on the presence of the compiler and knowing where the compiler is, uses up disk space, and requires the entire file to be re-written for even minor changes. JSPs on Tomcat are a good example of this. Ever noticed how slow the pages are the first time you look at them? That's because the JSP is being converted to plain old Java, written to a source file, compiled to a class file, loaded, and finally run.

The way around this would be to have a compiler built in. This would avoid executing an external process. Then if the compiler could output to a memory buffer instead of a class file, the results could be fed directly into a class loader, without having to use the disk.

However, a normal compiler would still expect to work on Java source code, which is just text. This still leaves the system rather inflexible, requiring a full recompile for every modification. Ideally, I'd want a compiler that could work directly on the Abstract Syntax Tree (AST) of Java. This would allow for easy and faster modifications. Since compilers have to generate an AST internally, this would also make compilation faster, since the text to AST conversion could be skipped.

If the compiler is to be operating directly on the AST, then where would the AST come from? Normally the AST would come from Java text, but I'm trying to avoid having to repeatedly convert text into an AST. I'd like to build a class from OWL, so should I be compiling that into an AST every time? OWL is more structured that text, but it would still be a lot of work to repeat on a dynamic system.

Ideally, it would be possible to convert each of these into an AST, but to then persistently store the AST for manipulation and compilation at any time. At this point I realized that I have an RDF database (after all, I'm doing modeling with OWL), and this would be perfect for storing the AST. This started to open up new possibilities.

The system would involve storing a complete Java AST in RDF. While the schema definition will take a little while, there is nothing hard in this (the schema is not required for storage, but rather to understand the structure in order to implement the API for the AST). To get things into the schema will require something that can compile Java source text into this AST. There are several open source Java compilers, along with SableCC definition files for parsing Java source, so this should be reasonably straightforward as well. Compiling OWL into the AST is a different matter, but appears to be possible with a set of inferencing rules.

The final step is the transformation of the AST into class files. This is a well documented procedure, though one that I've yet to learn properly. I can always leverage an open source compiler's implementation, but I will need a good understanding of this process if I'm to customize it accordingly. Besides, I've been meaning to read the Java spec for years.

Once the class binary is generated, a custom class loader will let this class be immediately loaded and instantiated. This could be very dynamic, allowing infinitely flexible new classes, with methods customized at runtime. Building these classes from semantics documents like OWL-S means that the system can dynamically reconfigure itself to manage what it discoveries about the world.

AST
Pivotal to all of this is the AST. James Gosling once gave an interview about Java 1.5 (I can't find it anymore) where he talked about the inclusion of an AST API. He was also working on a project called "Jackpot" which provided this API. Obviously this never eventuated for Java 1.5, though it's been suggested for Java 1.6. So if I can't use an official AST, what should I be using?

Should I go with the internal AST of a working compiler, like Kaffe? Should I go with the AST given to me by a SableCC parser? I figured that standards are a good thing here, so I went looking for what other people use.

The one AST that seems to have the best penetration comes from Eclipse. This bothers me for a few reasons. First, it is still slow and occasionally crashes on my Mac (though recently it's been getting faster and more stable). Second is the steep learning curve there seems to be to get into the internals of Eclipse. Finally, when I looked at the structure, it appears more complex than the ASTs I've seen elsewhere (maybe it's somehow more complete?).

Anyway, I haven't coded anything yet, so I'm still looking.

Possibilities
Having an architecture that stores the AST, compiling data into it, and then emitting it into binary, has several other advantages. Obviously, it becomes easy to modify code programmatically, and it is possible to have a single system that can compile multiple languages into the one AST format (Java, Jython, or annotated OWL).

This kind of system also makes it easy to work backwards from binary to text. Existing class files can be decomposed into their AST, and an AST can be converted into Java source text. Jad already converts from class to source code quite successfully (I'm guessing it must use an AST internally), so a precedent has been set here. However, this system would provide extra functionality. It could take a class file at run time, decompose it, update the existing code (for instance, adding instrumentation to methods), and then reload the class. I've heard of systems which do this sort of thing (particularly for adding instrumentation), but not with an API to control the modifications.

Ontologies
Normally, I'd be horrified at the idea of letting programmers out with such a powerful API, but the purpose here is not to permit programmers to perform ad hoc modifications, but rather to modify code according to the model described in an ontology.

I think this idea offers some interesting options for dynamically implementing models, and also for using an ontology to describe a program which can then be built automatically.

Ontologies are an area of research that is still moving quickly. It's hard to know exactly what this would contribute to it, but I think it would be quite useful.

Nic
For those who know me, I'd like to announce that Nicolas George Gearon was born at 3pm on Friday. Both he and his Mum are great (that's "Mom" for those of you in the US). Luc is still trying to work out why he is no longer the centre of attention.

Anne is a wonderful woman who never fails to impress me, especially so on days like Friday. However, she told me afterwards that if she ever does it again she'll use pain relief. I think that's fair enough. :-)

Family
The baby was due 8 days ago, so I'm a little distracted at the moment. The doctor will be inducing Anne tomorrow morning, so I should be a father again tomorrow. I wonder how much sleep I'll get tonight?

Grammar
I'm doing some interesting things with grammar parsing at work at the moment. I've looked at a few open source grammar parsers, but the one that I like the most is Link Grammar. It doesn't parse sentences into structural elements like other parsers, but instead looks for how individual words can link together. I've been pleasantly surprised at how easy it has been to use. I'm especially impressed at how easy it has been to modify the grammar to generally handle unusual situations.

I don't know how much I should be discussing it here. Just thought I'd mention it all the same.

Objective C and Cocoa
I'm still enjoying this, but it hasn't all been plain sailing.

The other night I had a problem with some simple code that would not work. It kept failing when I tried to call a method (or "send a message" in Objective C parlance, since it's not really a method call). According to the debugger, it could not find the method I was looking for. This object was being deserialized from the NIB file, so I tried instantiating a new instance of the object, and sending the message to that object instead. It still didn't work.

Trying to see everything running, I put a printf into the init method of the object. At this point I discovered that init was not being run when I instantiated the object the second time:

MyClass obj = [[MyClass alloc] init];

However, the init method was running just fine when the object came out of the NIB.

Out of frustration, I re-built a lot of this project (maybe something strange was happening in the NIB, and I have no way to work on that directly - only through limited APIs and the Interface Builder application). While doing this I discovered that I could crash the application during the init method on my class. If I have the line:

CGPoint size = CGPointMake(300.0, 400.0);

then I could guarantee a bus error when returning from init.

So this looks like I've smashed the stack somehow, though I can't see what I've done. I'm guessing that it has something to do with some element of Objective C message passing that I don't yet understand.

I fixed the problem by moving the set up code into another method, and not having an init in my class. It works, but I'd love to learn what the problem is.

In the meantime, this may have pointed me to another problem I've been having. DavidM still runs OS X v10.3 (Panther), rather than v10.4 (Tiger). I tried sending him a copy of one of my programs, but it crashes mysteriously. It's even more mysterious, as 3 other people running v10.4 all found that it worked fine. Now that I've seen the problems with init I've started wondering if this could have been the cause. To complicate matters, the program in question is multithreaded, so it's possible that there's a race between the new thread and init method, and it only shows up on v10.3. I've re-arranged the initialization in this code, so I'm keen to see how it works. (Unfortunately, I haven't seen David online since I made this change).

Compiler Hissy Fits
On other occasions I've had problems where the compiler wouldn't allow me to call init directly on a newly allocated object. So instead of the following (copied directly from the Apple Mutithreading documentation):

  NSAutoreleasePool* pool = [[NSAutoreleasePool alloc] init];

I was forced to split it up:

  NSAutoreleasePool* pool = [NSAutoreleasePool alloc];
  [pool init];

Now according to my understanding of Objective C, these are equivalent, so I don't understand why the first was failing.

However, I'm still learning. I'm even taking the time to read language documents now (taking the slow and methodical approach that I eschewed last week). Hopefully this extra insight will help.

Books
A couple of days ago I read the Slashdot review on Java Puzzles. I obviously have lots of spare time at the moment, so I decided to get a copy. I haven't been buying many technical books lately, as everything I want is usually online (and more up to date), but I love getting new books, so I thought I'd treat myself.

So far it's been a light read. I've only made it through the first couple of chapters, but I'm mostly getting the puzzles out. At least it's showing me that I do seem to understand the language spec reasonably well. However, I've picked it up from usage, experimentation, and knowledge of other language implementations. I really ought to read the spec one day! :-) (I keep meaning to. Maybe this will be my incentive?)

Work
Working on purely proprietary code makes it awkward to write about it in public. On top of that, it leaves my time a little too committed to do open source work.

So what is there to blog about? Well, there are still a few things, but having the baby due about now (yesterday, in fact) has been enough excuse to avoid writing. All the same, maybe I need to put some effort in. I keep finding myself wondering how I did something a few weeks ago, and the blog was supposed to help with that.

Cocoa
For a change of pace, I've spent my last two evenings learning Cocoa and Objective C. I'd avoided it before now because I didn't know Objective C, and wasn't sure I wanted to learn it when almost no other system makes use of it. Similarly, Cocoa is only available on the Mac, so there's no portability, unlike with QT or Java's Swing.

However, I like the Mac, and it's been bothering me that I haven't done any GUI coding for a while. I did some work with QT earlier this year, but it never looked quite right to me. (That would be more due to my knowledge of QT than anything else, as I've seen some attractive interfaces built with QT). I enjoy drawing mathematical shapes (yes, I could just learn more GNUPlot), and Quartz 2D sounds like fun. Since my after-hours work is all about fun, these seemed as good a set of reasons as any to try it.

I reasoned that there were two approaches I could try for this. The first was the traditional one of reading the docs and then applying my knowledge. That's very successful, and gives a very thorough knowledge of the subject material, but it also takes a long time. The other option is to learn just enough to get started, and then experiment with the documentation nearby. This technique leads to a much less thorough understanding of the system, and can result in accidentally taking a poor approach to a problem (since the best approach may not have been apparent). However, it gets you up and running faster. It's also more fun, because of the immediate feedback that you get. Given my criteria of fun and lack of time, the second approach seemed like a better idea.

What could I write that would be fun, and easy to write in my limited time in the evening? I decided to draw a simple curve which meets the equation:
x = sin(theta)
y = cos(3 * theta)

This looks like 3 cycles of a sine wave, wrapped around a clear cylinder. If I got it working, then I thought I should animate it (rotate the cylinder), and maybe introduce some controls to control the speed and number of cycles.

Objective C
I started with an ObjectiveC/Cocoa tutorial. This covered the development tools (Xcode and InterfaceBuilder) more than coding, but it covered a couple of the basics, and explained the basic syntax for method calls in Objective C (which I'd had trouble working out when I read example code). It turns out that calling methods like this is called "sending a message", which was the first hint I had of some of the "disconnected" nature of some of the features in the runtime system.

Once I thought I understood how to use Objective C, I left Xcode and tried my hand at some "Hello world" style programs on the command line. This worked pretty well, and helped me work out some more of the syntax of classes. I still have a LOT to learn (for instance, while I know @interface and @implementation, I haven't worked out what @protocol does). Still, I've always felt that if you can't do something like this manually from the command line, then you'll never be able to properly use the GUI tool that automates it for you.

The only thing I struggled with was the need to inherit from NSObject. This was needed for the alloc method, which does the heap allocation. I think I could implement this myself using malloc, but it was easier to use the Cocoa system instead.

It was only later that I discovered that ObjectiveC/Cocoa does memory reclamation when objects are no longer in use. An object called an allocator is used to assign memory for objects, and register them for later cleanup. The allocator is normally provided by the system, but it can also be set manually. This is essential to know when launching new threads, as there is no default allocator (it was while reading about Cocoa threads that I learned about automated object reclamation.

Other than memory management, there are two other features to ObjectiveC that stood out for me. The first is a kind of reflection which lets you analyze the available methods at runtime. This is interesting as it means that many objects don't need to inherit an interface, but just need to implement any required methods, ignoring anything not needed. This is the technique used by delegates.

The other feature is based on the numerous message passing mechanisms supported by the runtime. This reminds me of Windows messages, but seems to be built into the runtime (it was hard to tell what was part of the ObjectiveC runtime and what was provided by the Cocoa libraries). It can sometimes be difficult initializing an object with references to every other object that it needs to talk to, so a global message passing system can be really useful on occasion.

Observations
Over the years I've done GUI programming with the Windows API, MFC, Delphi, Xlib, Gnome/GTK, QT, Java AWT and Visual Basic. I'm certainly not an expert in any of these (at least, not anymore) but it's given me a taste of a number of approaches.

Just to provide some perspective on my impressions of Cocoa, here is a quick and dirty rundown on my impressions of each:

• Xlib: Lots of work to do anything at all. I wouldn't use it without a modern library to wrap it, to provide an attractive and consistent interface, and to provide useful modern widgets.
• Windows API: Less work than XLib, but there is still a lot of boilerplate code. Unfortunately a lot of Windows functionality now comes through object systems (COM+, ActiveX and more), and these can take a lot of code in C. All the same, once you have your head around the parts of the system you need, it's pretty easy to code. It can just take a lot of code to do some common tasks.
• MFC: At face value this wraps the Windows API, and implements a lot of common tasks, making it easier and quicker to write code. However, the action and messaging system is a dogs breakfast. If I want to respond to something, then do I implement a virtual function in a subclass, or do I add an entry to the message map? If it goes into the message map, then is there already a macro for it, or do I register for a particular windows message? It is also too dependent on Windows messages. Some basic functions are provided to wrap message (eg. setText), but most of the time the response to any message is to send a new message. Where's the OO in this? Sidestepping the class framework is possible when needed (reimplementing the message loop, for instance), but painful. Not to mention the difficulties working with MFC and ATL together. My biggest gripe was the need to learn a new API every week.
• Gnome/GTK: An extremely flexible system, running on Windows, OS X, and any X11 system. I've only used the standard C binding, but it comes with bindings for practically every useful language, including binary OO languages like C++. Glade also does a lot of the work in designing and implementing a GUI. My experience here was extremely limited, but I felt a little overwhelmed with options. The object design in C is good, but it does force certain development patterns on the programmer (this is probably a good thing, but feels constraining). The main reason I didn't go too far here was the sheer number of libraries needed to install on a non-Linux system to make your application work.
• QT: The big competitor to GTK/Gnome, running on the same systems. The GUI editor feels a little clunky, and I found it hard to make layout managers work exactly the way I wanted on all platforms. It feels more like MFC than any of the other frameworks here, but the messaging system is a lot cleaner and more consistent. I think Trolltech could have improved over MFC if they'd chosen to integrate with the C++ STL, but they haven't. This is demonstrate in their QString class, whose operator+(...) methods don't always do as you expect.
• Java AWT: Anyone using this interface will know why I stopped using it. Even Sun realized that the "lowest common denominator between windowing system" was never going to work, and moved on to Swing. However, it was easy enough to use, with all the work done with subclassing and methods to receive events. Layouts were limited, but their use was clearly documented. The main problem was the coding work required to set up the interface. This could be quite verbose, and needed to be run before the GUI was visible. I should just learn to use Swing, but I still hear about "clunkiness" problems. It's also buggy on OS X.4.
• Visual Basic: Back to being Windows only, but very easy to set up and code. The moment you try to do anything useful you discover that you're badly hamstrung and can't do much. It was necessary to call directly into the Windows API much more than it should have been. I used to write a lot of DLLs and call them from VB. Even worse was the propensity to launch background threads that you were never told about, leading to code that could fail on some invocations, but not on others.
• Delphi: I always found the syntax of this language to be too saccharine, but I can't deny the usefulness of this system. As easy to set up as VB, but with the power of MFC. I was particularly impressed when Borland made it fully compatible with C++ Builder. Most functionality is available in classes and methods rather than windows messages. Releasing Kylix for Linux was also a great move. It's a shame that there is no way to run it on any other system, such as OS X or Solaris. However, I can't afford the costs of Delphi, so I decided to leave it to the corporate types.

Overall, developing in Cocoa is really very nice. With object management and reflection, I almost felt like I was using Java, only easier. Everything I needed was in virtual methods, meaning I could do everything by subclassing. However, I decided that I wanted to shut down the application when the main window was closed, and I was concerned that I would therefore need to create a new subclass of NSWindow and use that as the class for the window. This is where I discovered delegates and some of the other notification mechanisms in use by ObjectiveC and Cocoa.

Delegates are objects given to a notifying object (such as an instance of NSWindow) which will be told when anything significant happens to the notifying object. The delegate object does not implement any interface and need not implement any methods except the ones it's interested in. This meant that I could simply have the main window inform an object of my choice when it was about to close, rather than having to introduce a new instance of a subclass of NSWindow, implementing a single method.

Of course, one of the nicest aspects of the system was access to subsystems like Quartz 2D. The resulting drawing had some nice extra features by default, such as antialiasing and buffering. The buffering became really apparent when I started animating the curve. I expected to do offscreen buffering and then start displaying completed images at the correct rate (there's even the NSViewAnimation class which will take these images and do the work for you), but Quartz did all the buffering for me. It even seemed to introduce some blur from one image to the next. The old drawing was wiped before the new one was created, so this appeared to be an intentional effect to make the animation seem smoother.

Some of the detractions include threads and the Interface Builder tool. Threads are managed with the NSThread class, but this really acts as a wrapper over some non-OO code that sets up a POSIX thread. The entry point to the thread can just be a method anywhere, rather than in a class specifically designed for threads. Also, the new thread has no default auto-release pool for objects, so to stop memory leaking it is important to set one up. This is trivial, but seems like the sort of thing that should have been done already. Also, operations like timers require a run loop to be processing, and again this has to be called manually. It works, but it's hardly OO.

The biggest problem with the Interface Builder was lack of documentation. The basics are covered in several tutorials, or when the help button is pressed, but I could not find advanced options mentioned anywhere. For instance, what does it mean to bind the value of a control to a user class instance? It seems to introduce a new dictionary with a key/value for the object's value, but where is this dictionary to be found?

To start with, the layout of the controls was difficult to work out, but I finally got it. I later found an obscure reference to it in some documents, but it still wasn't properly explained.

I also had several attempts to merge class files updated by Interface Builder with the originals in Xcode, but none were successful. In the end, I started generating new files from scratch, and manually copying the new elements over to the files in Xcode. I'm sure this can be made to work, but I didn't get it going. Again, documentation may be useful here.

Finally, on three separate occasions the Xcode IDE crashed when I clicked on a link in the documentation window. Normally the documentation is a separate program from the IDE, but Xcode can make requests of the documentation window all the time, so it all seems to be one program.

All these complaints aside, it is still a nice system. The Interface Builder lets you instantiate classes and then store them in the binary NIB file. These objects are then de-serialized at runtime, and are therefore instantiated on startup. This solves certain problems that can come up in object lifecycles, so I like it here. Connections between objects (between outlets and actions) are not clear unless the source object is selected in the GUI, but the concept is still nice. This is particularly useful as the objects at either end are extracted from the NIB, otherwise there would be no easy way to get references to either object to programmatically initialize this connection.

What do I think?
Weighing it up, this was one of the easier systems I've encoded on, and I was happy with the resulting program. For someone who likes coding (as opposed to a pure GUI configuration, as provided in VB or Delphi) then this is a nice and powerful system. The tool for building interfaces is very good, but doesn't integrate into the code well, which is why I deduct marks when comparing it to Delphi. I'm looking forward to writing more code in it. Maybe I'll check out why the curve isn't appearing when I do a print preview.

I commented the code, so I'm happy to give it to anyone who wants to see some example Cocoa code.