Projects
I've spent my last week and a bit on JNI. It was fun when I started (since I haven't done much C lately), but ultimately tedious. After all, I'm just writing a wrapper around someone else's library. Sure, there are some cute challenges, but it's not quantum physics (that reminds me... I really need to get back to postgrad physics at some point. I should finish this Masters).

In the meantime, I've been looking at a project for web services to keep my mind fresh. Or more precisely, I've been looking at another project when I'm not working or looking after the boys (yes, I'm averaging below 5 hours sleep per night). It's a project I've mentioned on and off for a while now, but I've decided to do something about it.

The principles of the project came about through a combination of several factors. I've had some experience with an application server (where I wrote the JMX code). I've also been working with OWL, from a theoretical point of view, an inferencing perspective, and using it for modeling in several systems. I've been working with UML modeling, and writing an OCL interpreter to manage those models at runtime. I'm also an engineer who really likes bit-banging as opposed to all this high-level abstraction stuff (though the high-level stuff does have its charms). Finally, I've had several interesting conversations with people who've helped me crystallize what I'm trying to achieve.

The Idea
The idea is to use OWL to describe a class that can be instantiated in Java. There are several aspects of OWL that don't make it ideal for this kind of modeling, but it is still possible to use it. This is demonstrated by the ability to map most UML features into OWL. Fortunately, the flexibility of RDF allows almost any conceivable type of annotation to be added to the class, filling in any areas where OWL is not up to the task.

What would be the point of this? Well the first things that comes to mind are Web Services (thanks to David for this suggestion). Currently, services can described in OWL-S. If a client does not know about a described service, then it can always try to model it (this technology is still in development). However, why simulate the model, when you can instantiate a class that meets the description of the model? This would perform better, and offer much more flexibility to the client system.

But How?
One approach for this would be to convert the OWL class into some implementing Java source code, write it to disk, and convert it into a class file with javac. I've never liked this approach. It is very slow, relies on the presence of the compiler and knowing where the compiler is, uses up disk space, and requires the entire file to be re-written for even minor changes. JSPs on Tomcat are a good example of this. Ever noticed how slow the pages are the first time you look at them? That's because the JSP is being converted to plain old Java, written to a source file, compiled to a class file, loaded, and finally run.

The way around this would be to have a compiler built in. This would avoid executing an external process. Then if the compiler could output to a memory buffer instead of a class file, the results could be fed directly into a class loader, without having to use the disk.

However, a normal compiler would still expect to work on Java source code, which is just text. This still leaves the system rather inflexible, requiring a full recompile for every modification. Ideally, I'd want a compiler that could work directly on the Abstract Syntax Tree (AST) of Java. This would allow for easy and faster modifications. Since compilers have to generate an AST internally, this would also make compilation faster, since the text to AST conversion could be skipped.

If the compiler is to be operating directly on the AST, then where would the AST come from? Normally the AST would come from Java text, but I'm trying to avoid having to repeatedly convert text into an AST. I'd like to build a class from OWL, so should I be compiling that into an AST every time? OWL is more structured that text, but it would still be a lot of work to repeat on a dynamic system.

Ideally, it would be possible to convert each of these into an AST, but to then persistently store the AST for manipulation and compilation at any time. At this point I realized that I have an RDF database (after all, I'm doing modeling with OWL), and this would be perfect for storing the AST. This started to open up new possibilities.

The system would involve storing a complete Java AST in RDF. While the schema definition will take a little while, there is nothing hard in this (the schema is not required for storage, but rather to understand the structure in order to implement the API for the AST). To get things into the schema will require something that can compile Java source text into this AST. There are several open source Java compilers, along with SableCC definition files for parsing Java source, so this should be reasonably straightforward as well. Compiling OWL into the AST is a different matter, but appears to be possible with a set of inferencing rules.

The final step is the transformation of the AST into class files. This is a well documented procedure, though one that I've yet to learn properly. I can always leverage an open source compiler's implementation, but I will need a good understanding of this process if I'm to customize it accordingly. Besides, I've been meaning to read the Java spec for years.

Once the class binary is generated, a custom class loader will let this class be immediately loaded and instantiated. This could be very dynamic, allowing infinitely flexible new classes, with methods customized at runtime. Building these classes from semantics documents like OWL-S means that the system can dynamically reconfigure itself to manage what it discoveries about the world.

AST
Pivotal to all of this is the AST. James Gosling once gave an interview about Java 1.5 (I can't find it anymore) where he talked about the inclusion of an AST API. He was also working on a project called "Jackpot" which provided this API. Obviously this never eventuated for Java 1.5, though it's been suggested for Java 1.6. So if I can't use an official AST, what should I be using?

Should I go with the internal AST of a working compiler, like Kaffe? Should I go with the AST given to me by a SableCC parser? I figured that standards are a good thing here, so I went looking for what other people use.

The one AST that seems to have the best penetration comes from Eclipse. This bothers me for a few reasons. First, it is still slow and occasionally crashes on my Mac (though recently it's been getting faster and more stable). Second is the steep learning curve there seems to be to get into the internals of Eclipse. Finally, when I looked at the structure, it appears more complex than the ASTs I've seen elsewhere (maybe it's somehow more complete?).

Anyway, I haven't coded anything yet, so I'm still looking.

Possibilities
Having an architecture that stores the AST, compiling data into it, and then emitting it into binary, has several other advantages. Obviously, it becomes easy to modify code programmatically, and it is possible to have a single system that can compile multiple languages into the one AST format (Java, Jython, or annotated OWL).

This kind of system also makes it easy to work backwards from binary to text. Existing class files can be decomposed into their AST, and an AST can be converted into Java source text. Jad already converts from class to source code quite successfully (I'm guessing it must use an AST internally), so a precedent has been set here. However, this system would provide extra functionality. It could take a class file at run time, decompose it, update the existing code (for instance, adding instrumentation to methods), and then reload the class. I've heard of systems which do this sort of thing (particularly for adding instrumentation), but not with an API to control the modifications.

Ontologies
Normally, I'd be horrified at the idea of letting programmers out with such a powerful API, but the purpose here is not to permit programmers to perform ad hoc modifications, but rather to modify code according to the model described in an ontology.

I think this idea offers some interesting options for dynamically implementing models, and also for using an ontology to describe a program which can then be built automatically.

Ontologies are an area of research that is still moving quickly. It's hard to know exactly what this would contribute to it, but I think it would be quite useful.

Nic
For those who know me, I'd like to announce that Nicolas George Gearon was born at 3pm on Friday. Both he and his Mum are great (that's "Mom" for those of you in the US). Luc is still trying to work out why he is no longer the centre of attention.

Anne is a wonderful woman who never fails to impress me, especially so on days like Friday. However, she told me afterwards that if she ever does it again she'll use pain relief. I think that's fair enough. :-)

Family
The baby was due 8 days ago, so I'm a little distracted at the moment. The doctor will be inducing Anne tomorrow morning, so I should be a father again tomorrow. I wonder how much sleep I'll get tonight?

Grammar
I'm doing some interesting things with grammar parsing at work at the moment. I've looked at a few open source grammar parsers, but the one that I like the most is Link Grammar. It doesn't parse sentences into structural elements like other parsers, but instead looks for how individual words can link together. I've been pleasantly surprised at how easy it has been to use. I'm especially impressed at how easy it has been to modify the grammar to generally handle unusual situations.

I don't know how much I should be discussing it here. Just thought I'd mention it all the same.

Objective C and Cocoa
I'm still enjoying this, but it hasn't all been plain sailing.

The other night I had a problem with some simple code that would not work. It kept failing when I tried to call a method (or "send a message" in Objective C parlance, since it's not really a method call). According to the debugger, it could not find the method I was looking for. This object was being deserialized from the NIB file, so I tried instantiating a new instance of the object, and sending the message to that object instead. It still didn't work.

Trying to see everything running, I put a printf into the init method of the object. At this point I discovered that init was not being run when I instantiated the object the second time:

MyClass obj = [[MyClass alloc] init];

However, the init method was running just fine when the object came out of the NIB.

Out of frustration, I re-built a lot of this project (maybe something strange was happening in the NIB, and I have no way to work on that directly - only through limited APIs and the Interface Builder application). While doing this I discovered that I could crash the application during the init method on my class. If I have the line:

CGPoint size = CGPointMake(300.0, 400.0);

then I could guarantee a bus error when returning from init.

So this looks like I've smashed the stack somehow, though I can't see what I've done. I'm guessing that it has something to do with some element of Objective C message passing that I don't yet understand.

I fixed the problem by moving the set up code into another method, and not having an init in my class. It works, but I'd love to learn what the problem is.

In the meantime, this may have pointed me to another problem I've been having. DavidM still runs OS X v10.3 (Panther), rather than v10.4 (Tiger). I tried sending him a copy of one of my programs, but it crashes mysteriously. It's even more mysterious, as 3 other people running v10.4 all found that it worked fine. Now that I've seen the problems with init I've started wondering if this could have been the cause. To complicate matters, the program in question is multithreaded, so it's possible that there's a race between the new thread and init method, and it only shows up on v10.3. I've re-arranged the initialization in this code, so I'm keen to see how it works. (Unfortunately, I haven't seen David online since I made this change).

Compiler Hissy Fits
On other occasions I've had problems where the compiler wouldn't allow me to call init directly on a newly allocated object. So instead of the following (copied directly from the Apple Mutithreading documentation):

  NSAutoreleasePool* pool = [[NSAutoreleasePool alloc] init];

I was forced to split it up:

  NSAutoreleasePool* pool = [NSAutoreleasePool alloc];
  [pool init];

Now according to my understanding of Objective C, these are equivalent, so I don't understand why the first was failing.

However, I'm still learning. I'm even taking the time to read language documents now (taking the slow and methodical approach that I eschewed last week). Hopefully this extra insight will help.

Books
A couple of days ago I read the Slashdot review on Java Puzzles. I obviously have lots of spare time at the moment, so I decided to get a copy. I haven't been buying many technical books lately, as everything I want is usually online (and more up to date), but I love getting new books, so I thought I'd treat myself.

So far it's been a light read. I've only made it through the first couple of chapters, but I'm mostly getting the puzzles out. At least it's showing me that I do seem to understand the language spec reasonably well. However, I've picked it up from usage, experimentation, and knowledge of other language implementations. I really ought to read the spec one day! :-) (I keep meaning to. Maybe this will be my incentive?)

Work
Working on purely proprietary code makes it awkward to write about it in public. On top of that, it leaves my time a little too committed to do open source work.

So what is there to blog about? Well, there are still a few things, but having the baby due about now (yesterday, in fact) has been enough excuse to avoid writing. All the same, maybe I need to put some effort in. I keep finding myself wondering how I did something a few weeks ago, and the blog was supposed to help with that.

Cocoa
For a change of pace, I've spent my last two evenings learning Cocoa and Objective C. I'd avoided it before now because I didn't know Objective C, and wasn't sure I wanted to learn it when almost no other system makes use of it. Similarly, Cocoa is only available on the Mac, so there's no portability, unlike with QT or Java's Swing.

However, I like the Mac, and it's been bothering me that I haven't done any GUI coding for a while. I did some work with QT earlier this year, but it never looked quite right to me. (That would be more due to my knowledge of QT than anything else, as I've seen some attractive interfaces built with QT). I enjoy drawing mathematical shapes (yes, I could just learn more GNUPlot), and Quartz 2D sounds like fun. Since my after-hours work is all about fun, these seemed as good a set of reasons as any to try it.

I reasoned that there were two approaches I could try for this. The first was the traditional one of reading the docs and then applying my knowledge. That's very successful, and gives a very thorough knowledge of the subject material, but it also takes a long time. The other option is to learn just enough to get started, and then experiment with the documentation nearby. This technique leads to a much less thorough understanding of the system, and can result in accidentally taking a poor approach to a problem (since the best approach may not have been apparent). However, it gets you up and running faster. It's also more fun, because of the immediate feedback that you get. Given my criteria of fun and lack of time, the second approach seemed like a better idea.

What could I write that would be fun, and easy to write in my limited time in the evening? I decided to draw a simple curve which meets the equation:
x = sin(theta)
y = cos(3 * theta)

This looks like 3 cycles of a sine wave, wrapped around a clear cylinder. If I got it working, then I thought I should animate it (rotate the cylinder), and maybe introduce some controls to control the speed and number of cycles.

Objective C
I started with an ObjectiveC/Cocoa tutorial. This covered the development tools (Xcode and InterfaceBuilder) more than coding, but it covered a couple of the basics, and explained the basic syntax for method calls in Objective C (which I'd had trouble working out when I read example code). It turns out that calling methods like this is called "sending a message", which was the first hint I had of some of the "disconnected" nature of some of the features in the runtime system.

Once I thought I understood how to use Objective C, I left Xcode and tried my hand at some "Hello world" style programs on the command line. This worked pretty well, and helped me work out some more of the syntax of classes. I still have a LOT to learn (for instance, while I know @interface and @implementation, I haven't worked out what @protocol does). Still, I've always felt that if you can't do something like this manually from the command line, then you'll never be able to properly use the GUI tool that automates it for you.

The only thing I struggled with was the need to inherit from NSObject. This was needed for the alloc method, which does the heap allocation. I think I could implement this myself using malloc, but it was easier to use the Cocoa system instead.

It was only later that I discovered that ObjectiveC/Cocoa does memory reclamation when objects are no longer in use. An object called an allocator is used to assign memory for objects, and register them for later cleanup. The allocator is normally provided by the system, but it can also be set manually. This is essential to know when launching new threads, as there is no default allocator (it was while reading about Cocoa threads that I learned about automated object reclamation.

Other than memory management, there are two other features to ObjectiveC that stood out for me. The first is a kind of reflection which lets you analyze the available methods at runtime. This is interesting as it means that many objects don't need to inherit an interface, but just need to implement any required methods, ignoring anything not needed. This is the technique used by delegates.

The other feature is based on the numerous message passing mechanisms supported by the runtime. This reminds me of Windows messages, but seems to be built into the runtime (it was hard to tell what was part of the ObjectiveC runtime and what was provided by the Cocoa libraries). It can sometimes be difficult initializing an object with references to every other object that it needs to talk to, so a global message passing system can be really useful on occasion.

Observations
Over the years I've done GUI programming with the Windows API, MFC, Delphi, Xlib, Gnome/GTK, QT, Java AWT and Visual Basic. I'm certainly not an expert in any of these (at least, not anymore) but it's given me a taste of a number of approaches.

Just to provide some perspective on my impressions of Cocoa, here is a quick and dirty rundown on my impressions of each:

• Xlib: Lots of work to do anything at all. I wouldn't use it without a modern library to wrap it, to provide an attractive and consistent interface, and to provide useful modern widgets.
• Windows API: Less work than XLib, but there is still a lot of boilerplate code. Unfortunately a lot of Windows functionality now comes through object systems (COM+, ActiveX and more), and these can take a lot of code in C. All the same, once you have your head around the parts of the system you need, it's pretty easy to code. It can just take a lot of code to do some common tasks.
• MFC: At face value this wraps the Windows API, and implements a lot of common tasks, making it easier and quicker to write code. However, the action and messaging system is a dogs breakfast. If I want to respond to something, then do I implement a virtual function in a subclass, or do I add an entry to the message map? If it goes into the message map, then is there already a macro for it, or do I register for a particular windows message? It is also too dependent on Windows messages. Some basic functions are provided to wrap message (eg. setText), but most of the time the response to any message is to send a new message. Where's the OO in this? Sidestepping the class framework is possible when needed (reimplementing the message loop, for instance), but painful. Not to mention the difficulties working with MFC and ATL together. My biggest gripe was the need to learn a new API every week.
• Gnome/GTK: An extremely flexible system, running on Windows, OS X, and any X11 system. I've only used the standard C binding, but it comes with bindings for practically every useful language, including binary OO languages like C++. Glade also does a lot of the work in designing and implementing a GUI. My experience here was extremely limited, but I felt a little overwhelmed with options. The object design in C is good, but it does force certain development patterns on the programmer (this is probably a good thing, but feels constraining). The main reason I didn't go too far here was the sheer number of libraries needed to install on a non-Linux system to make your application work.
• QT: The big competitor to GTK/Gnome, running on the same systems. The GUI editor feels a little clunky, and I found it hard to make layout managers work exactly the way I wanted on all platforms. It feels more like MFC than any of the other frameworks here, but the messaging system is a lot cleaner and more consistent. I think Trolltech could have improved over MFC if they'd chosen to integrate with the C++ STL, but they haven't. This is demonstrate in their QString class, whose operator+(...) methods don't always do as you expect.
• Java AWT: Anyone using this interface will know why I stopped using it. Even Sun realized that the "lowest common denominator between windowing system" was never going to work, and moved on to Swing. However, it was easy enough to use, with all the work done with subclassing and methods to receive events. Layouts were limited, but their use was clearly documented. The main problem was the coding work required to set up the interface. This could be quite verbose, and needed to be run before the GUI was visible. I should just learn to use Swing, but I still hear about "clunkiness" problems. It's also buggy on OS X.4.
• Visual Basic: Back to being Windows only, but very easy to set up and code. The moment you try to do anything useful you discover that you're badly hamstrung and can't do much. It was necessary to call directly into the Windows API much more than it should have been. I used to write a lot of DLLs and call them from VB. Even worse was the propensity to launch background threads that you were never told about, leading to code that could fail on some invocations, but not on others.
• Delphi: I always found the syntax of this language to be too saccharine, but I can't deny the usefulness of this system. As easy to set up as VB, but with the power of MFC. I was particularly impressed when Borland made it fully compatible with C++ Builder. Most functionality is available in classes and methods rather than windows messages. Releasing Kylix for Linux was also a great move. It's a shame that there is no way to run it on any other system, such as OS X or Solaris. However, I can't afford the costs of Delphi, so I decided to leave it to the corporate types.

Overall, developing in Cocoa is really very nice. With object management and reflection, I almost felt like I was using Java, only easier. Everything I needed was in virtual methods, meaning I could do everything by subclassing. However, I decided that I wanted to shut down the application when the main window was closed, and I was concerned that I would therefore need to create a new subclass of NSWindow and use that as the class for the window. This is where I discovered delegates and some of the other notification mechanisms in use by ObjectiveC and Cocoa.

Delegates are objects given to a notifying object (such as an instance of NSWindow) which will be told when anything significant happens to the notifying object. The delegate object does not implement any interface and need not implement any methods except the ones it's interested in. This meant that I could simply have the main window inform an object of my choice when it was about to close, rather than having to introduce a new instance of a subclass of NSWindow, implementing a single method.

Of course, one of the nicest aspects of the system was access to subsystems like Quartz 2D. The resulting drawing had some nice extra features by default, such as antialiasing and buffering. The buffering became really apparent when I started animating the curve. I expected to do offscreen buffering and then start displaying completed images at the correct rate (there's even the NSViewAnimation class which will take these images and do the work for you), but Quartz did all the buffering for me. It even seemed to introduce some blur from one image to the next. The old drawing was wiped before the new one was created, so this appeared to be an intentional effect to make the animation seem smoother.

Some of the detractions include threads and the Interface Builder tool. Threads are managed with the NSThread class, but this really acts as a wrapper over some non-OO code that sets up a POSIX thread. The entry point to the thread can just be a method anywhere, rather than in a class specifically designed for threads. Also, the new thread has no default auto-release pool for objects, so to stop memory leaking it is important to set one up. This is trivial, but seems like the sort of thing that should have been done already. Also, operations like timers require a run loop to be processing, and again this has to be called manually. It works, but it's hardly OO.

The biggest problem with the Interface Builder was lack of documentation. The basics are covered in several tutorials, or when the help button is pressed, but I could not find advanced options mentioned anywhere. For instance, what does it mean to bind the value of a control to a user class instance? It seems to introduce a new dictionary with a key/value for the object's value, but where is this dictionary to be found?

To start with, the layout of the controls was difficult to work out, but I finally got it. I later found an obscure reference to it in some documents, but it still wasn't properly explained.

I also had several attempts to merge class files updated by Interface Builder with the originals in Xcode, but none were successful. In the end, I started generating new files from scratch, and manually copying the new elements over to the files in Xcode. I'm sure this can be made to work, but I didn't get it going. Again, documentation may be useful here.

Finally, on three separate occasions the Xcode IDE crashed when I clicked on a link in the documentation window. Normally the documentation is a separate program from the IDE, but Xcode can make requests of the documentation window all the time, so it all seems to be one program.

All these complaints aside, it is still a nice system. The Interface Builder lets you instantiate classes and then store them in the binary NIB file. These objects are then de-serialized at runtime, and are therefore instantiated on startup. This solves certain problems that can come up in object lifecycles, so I like it here. Connections between objects (between outlets and actions) are not clear unless the source object is selected in the GUI, but the concept is still nice. This is particularly useful as the objects at either end are extracted from the NIB, otherwise there would be no easy way to get references to either object to programmatically initialize this connection.

What do I think?
Weighing it up, this was one of the easier systems I've encoded on, and I was happy with the resulting program. For someone who likes coding (as opposed to a pure GUI configuration, as provided in VB or Delphi) then this is a nice and powerful system. The tool for building interfaces is very good, but doesn't integrate into the code well, which is why I deduct marks when comparing it to Delphi. I'm looking forward to writing more code in it. Maybe I'll check out why the curve isn't appearing when I do a print preview.

I commented the code, so I'm happy to give it to anyone who wants to see some example Cocoa code.