Ovidiu Predescu's Weblog

java.nio.channels.FileChannel is not selectable?

Maybe I'm missing something, but I just can't figure this out. With the Java NIO it appears there is no way you can use selectors to asynchronously listen for new data to read on stdin, or any other files for that matter!

I want to write a simple program that opens a serial device file (/dev/ttyS0 for example) on Unix (Linux or MacOS X), and listens to events on the associated file descriptor. These events are bytes written by a physical device connected on serial port. I also want the application to listen on stdin, for characters I may type to change the internal state of the application.

With the Unix select() you can do this easily by passing the two file descriptors to the system call. The Java API however chose to hide this ability, and you can only do selects on sockets. This is really stupid and plain wrong if you ask me, as it basically means I need to write the multithreaded code to handle this simple application.

How could the bunch of Unix heads at Sun that designed the NIO API allow such a thing to happen? Somebody tell me I'm wrong, please!

Update (April 5, 2005): Java is just the wrong language for my application. I've ported everything to Python, and implemented the above functionality in just few lines. Sometimes too many abstractions are just wrong, as in this case. Python exposes everything I need, including device characteristics (serial line speed, parity etc.). The perfect tool for the job!

Languages with macros

Nemerle, a language using a syntax similar to C#, has some very interesting support for syntactic macros. It basically allows programmers to extend the syntax of the language with new constructions, not already implemented in the language.

The sample macros they have as examples are for loops and a printf-like function with variable number of arguments. This shows once more why syntactic macros are a good thing to have in a language: there's no need for language developers to implement these features at the language level! The printf as well as other new syntactic-sugars in JDK 1.5 could have been implemented this way.

Last year, during a talk James Gosling gave at Google, I had the opportunity to ask him about macros in Java. His response was along the lines of "we don't need macros, since they are so hard to understand." Needless to say, I was really disappointed by his response.

Interestingly enough, the Nemerle developers seem to have received a grant from Microsoft. Quite interesting, I wonder how long will it take to see C# with macros.

XSLT whines

Bertrand writes:

I cannot agree more, although I very much like the data-driven execution model in XSLT. I also like the XPath data accessing model.

The top two things I don't like about XSLT are:

• The XML syntax, which makes programs very verbose.
• The extensibility mechanism is language agnostic, but it makes it difficult to write complex XSLT/Java programs that integrate nicely.

From this perspective, the SXML processing tools available for Scheme are a very nice integration of XML in a programming language. In SXML the XML processing is described as a real Scheme program that has access to all the library functions for traversing and accessing the XML tree. I wish Java had something of a similar simplicity, but instead it got all the DOM and SAX crap, and no portable XPath library implementation.

What is even scarier is that XSLT 2.0 is moving towards becoming a full programming language with an XML syntax.

One thing I started to pay more attention to is XQuery, and the excellent implementation done in Qexo by Per Bothner. XQuery promises to solve some of the XSLT verbosity I mentioned above. The processing model is closer to functional programming, although it does appear to loose the data-driven processing model I like in XSLT.

J2EE security book

Pankaj Kumar, my old colleague at HP had his book on J2EE security finally published at Prentice Hall. The companion Web site, j2ee-security.net, has some more insights of the book, including a table of contents. The book promises to be a good introduction to the security of Web services.

Pankaj spent a lot of time working on this book, and I always admired his ability to concentrate on doing the work, even when our group at HP went through some rough times. I hope all this work was worth the effort!

JavaOne pictures

I've uploaded some of the pictures I made at JavaOne in my picture gallery. Appologies for the poor quality, I didn't get too close to the speakers in most of the cases. I also updated the previous entries to contain a relevant picture.

JavaOne - compiling programming languages to the JVM

Per Bothner's talk on Kawa is about to start. Per worked on various things over the year. He used to work for Cygnus, before being acquired by RedHat; he worked on GCC and many other tools.

17:16 It started. Apparently Per is the only presenter not affiliated with any company, which an almost unheard of at JavaOne. The attendance is pretty light, it's either too late or people don't care about languages other than Java.

What do you do when you need a higher language than Java? Well, you can write an interpreter. However if you do repetitive computations, it can get pretty slow. Another approach is to compile the program in your language to Java source code.

The best approach is to compile directly to in-memory Java bytecodes. Per makes the interesting assertion that bytecodes are more general than Java source: you actually have goto statements.

Kawa was written while at Cygnus in 1996 and is a GNU project, with a more liberal license than GPL. Kawa can be run interactively from the command line, it can be compiled to a program file. Languages implemented in Kawa: Scheme, XQuery, XSLT, Emacs Lisp etc.

Short introduction to Scheme, an "impure" functional language because of assignment. You run Kawa by doing java kawa.repl and you get the interactive prompt. It supports big integers.You can write applets in Scheme's Kawa.

Another language supported is Common Lisp. Guy Steele was instrumental in the Scheme, Common Lisp and Java languages.

Emacs Lisp: "Emacs is still the most powerful text editor". Kawa compiles Elisp to Java bytecodes. Goal is modern Emacs re-implementation that can efficiently use existing Emacs packages. Uses Swing to represent and display text. A nice JEmacs screenshot, unfortunately not many people actually contribute to it.

XQuery is a very high-level language used for querying, combining and generating XML-like data sets. It is a superset of XPath. Kawa supports XQuery with Qexo, which is missing some features, but still very useful. Some example of XQuery to generate an HTML fragment: it uses HTML markup and XQuery syntax to generate the output page. The example can generate XHTML 1.0, HTML, Scheme from the same file. The example XQuery program can be compiled to a Java class with a main, or to a servlet which can be deployed on a servlet container.

XQuery can be considered as an alternative to JSP. An XQuery program can also be compiled to a CGI program, not very useful these days however. You can embed the XQuery engine in a Java program and take advantage of its power.

Next language shown is XSLT. The Kawa implementation compiles an XSLT stylesheet into a Java class. The project is incomplete, but it's a useful example.

BRL is Beautiful Report Language, a template language much like JSP. Instead of embedding Java, you embed Scheme. KRL - Kawa Report Language - is Per's implementation. The language uses square brackets to embed Scheme code. You can embed such code within HTML tags.

Nice is a strongly typed language with multi-methods, parametric types, anonymous functions, tuples and multiple implementation. KRL and Nice were both written by other people than Per.

Implementation. Each language is a subclass of Interpreter. Each Interpreter uses a Lexer to parse an expression or a program file. The result is an Excpression instance. There are many subclasses of Expression. Once you have an Expression object, you call the compile() method to compile the script. This method takes two arguments, a Compilation object for managing the state, and a Target object for specifying where to leave the result, usually the JVM's stack.

JavaOne - XML parsing (with StAX)

Even though I've heard about XML pull parsers, I haven't had time to play with them. From the abstract, this session seems to be a good introduction to the technology. The presenters, Ping Guo, Mark Scardina and K Karun are from Oracle.

I'm actually interested in more higher level XML processors, similar to XSLT, but that don't require a DOM tree representation of the input XML document. I looked at STX, Streaming Transformations for XML, and Joost, an open-source Java implementation of STX. The idea is intruiging and this particular implementation seems to be very clean. The CVS version seems to also have support for hooking up with the Java classes running outside the processor.

The Joost STX implementation uses a SAX parser to do the work, which makes it difficult to abort the processing of large XML documents. This could come in handy when in the middle of the processing, the program discovers it has nothing else to do. This would be a very useful feature for a streaming XML processor. StAX could solve this problem.

14:45 It started. Ping is the lead developer and the presenter of this session, K is an expert group member of JSR 173, and Mark is a manager. StAX is the subject of JSR 173.

Why do such presentations have to start with what is XML? Or examples of what an XML document is? I find this a waste of time, especially for a session marked as "intermediary" complexity. You would expect the audience to know these things.

Next is an introduction to different XML parsing methodologies. She introduces DOM now. She jumps to JAXP now, the Java API for accessing a DOM parser. She now presents a DOM parser example, which I found to be flawed because it uses getElementsByName(). The reason why this is bad, especially in the context of an input document conforming to a well known DTD, is because this method will traverse all the nodes in the document. She goes into explaining the cons of using DOM.

15:01 She now introduces SAX, explaining the model and giving a small example.

15:09 Finally, the meat of the talk, StAX. This new parsing technology uses a different event model. It has an XMLEvent abstract event, with StartDocument, StartElement etc. concrete events. The interface with the parse is very similar to JAXP. The parsing model is different from SAX: the application tells the parser to get the next event and the parser returns the concreate event objects.

The cursor style example

while (reader.hasNext()) {
  int eventType = reader.next();
  if (eventType == XMLEvent.START_ELEMENT && reader.getLocalName().equals("title")) {
    reader.next();
    println(reader.getText());
  }
}

The next example (iterator style) she presents asks the parser to return the actual event object, which may be easier to program but uses more memory because the parser has to create the event object (an actual XMLEvent instance).

The nice part about StAX is that you can process multiple XML documents at the same time, by simply sucking data from multiple parsers.

She now jumps into Web Services and JAX-RPC. This seems to be going over the scope of this talk, so I'll head out of the talk. StAX and XML pull-parsers seem to be a really good idea. I need to check them out.

JavaOne - Design patterns for high-load Web sites

I came in really late at this one, as I had to eat something and post the previous Weblog entries.

This session looks like a product presentation. It's a Windows RAD application for developing Web sites and Web services using Struts and presumable other J2EE frameworks. I didn't get the presenter's name or the company he works for.

JavaOne - More programming puzzles

This session promises to be a lot of fun. In the past years, I missed similar presentations considering them entry level, only to hear from colleagues that attended that the sessions were really good. The session is presented by Joshua Bloch and Neal Gafter of Sun Microsystems. After this session I need to get some food, I'm really hungry.

12:15pm It started. The first problem is "Random Behavior".

public class RandomSet {
  public static  void main(String[] args) {
    Set s = new HashSet();
    for (int i = 0; i < 100; i++)
      s.add(randomInteger());
    System.out.println(s.size());
  }

  private static Integer randomInteger() {
    return new Integer(new Random().nextInt());
  }
}

What is the size of the set printed at the end: 1, 50 or 100? Because the Random is initialized each time in randomInteger(), and that is always initialized with the current time of day in miliseconds, on fast machines the seed is pretty much the same. By sharing the Random object, you can actually get real random numbers.

Second problem is "Making a Hash of It".

public class Name {
  private String first, last;

  public Name(String first, String last) {
    if (first == null || last == null)
      throw new NullPointerException();
    this.first == first; this.last = last;
  }
  public boolean equals(Name o) {
    return first.equals(o.first) && last.equals(o.last);
  }

  public int hashCode() {
    return 31 * first.hashCode() + last.hashCode();
  {

  public static void main(String[] args)
    Set S = new HashSet();
    s.add(new Name("Mickey", "Mouse"));
    System.ouyt.println(s.contains(new Name("Mickey", "Mouse"));
  }
}

What does this program output, true or false? Because equals() takes as argument a Name and not an Object, the method is not actually overriding equals(Object), but overloads equals(). Thus the method is never called, so the program always returns false. The solution is to change the signature of equals() to take an Object as argument.

Next problem is "Ping Pong":

class PingPong {
  public static synchronized void main(String[] a) {
    Thread t = new Thread() {
      public void run() {
        pong();
      }
    };

    t.run();
    System.out.println("Ping");
  }

  static synchronized void pong() {
    System.out.println("Pong");
  }
}

What is the output of this program? PingPong, PongPing or it varies?

The answer is PongPing, because by the time pong() executes the class already holds the lock. Well, if the programmer actually wanted to start the thread, he should have written t.start() instead of t.run(). In that case the output would be PingPong. The issue is that Thread implements Runnable, which it shouldn't because it's difficult to catch these problems.

Next problem is "Reflection Infection":

public class Reflector {
  public static void main(String[] args) throws Exception {
    Set s = new HashSet();
    s.add("foo");
    Iterator i = s.iterator();
    Method m = i.getClass().getMethod("hasNext", new Class[0]);
    System.out.println(m.invoke(i, new Object[0]));
  }
}

Won't compile? true? Throws exception? None of the above?

Most people think it will throw an exception, which is correct: attempt to invoke a method on a private class. The problem is that i.getClass() is not an Iterator, which is an interface, but a private class of the JVM. The correct way to do it is rewrite it to Iterator.class.getMethod() instead.

Next problem is "String Cheese":

public class StringCheese {
  public static void main(String[] args) throws Exception {
    byte b[] = new byte[256];
    for (int i = 0; i < 256; i++)
      b[i] = (byte)i;
    String str = new String(b);
    for (int i = 0; i < str.length(); i++)
      System.out.print((int)str,charAt(i) + " ");
  }
}

The numbers from 0 to 255? The number from 0 to 127 then -128 to -1? It varies? None of the above?

The answer is it varies because the sequence depends on the default charset, which depends on OS and locale. The problem is String(byte[] bytes) which constructs a String according to the default charset and locale. To fix it you have to explicitly specify the charset new String(b, "ISO-8553-1")

Next problem is "All Strung Out":

public class Puzzle {
  public static void main(String[] args) {
    String s = new String("bla");
    System.out.println(s);
  }
}

class String {
  java.lang.String s;

  public String(java.lang.String s) {
    this.s = s;
  }

  public java.lang.String toString() {
    return s;
  }
}

Won't compile? blah? Throws an exception? Other?

It throws an exception at runtime. NoSuchMethodError is throwsn because Puzzling is missing a main method. The String argument of main() is the String class defined by us. To fix the problem you have to declare java.lang.String or rename the our String class to MyString. Moral: don't reuse names by hiding, shadowing or overloading.

Next problem is "Elvis Lives":

public class Elvis {
  public static final Elvis INSTANCE = new Elvis();
  private final int beltSize;

  private static final int CURRENT_YEAR = Calendar.getInstance().get(Calendar.YEAR);

  private Elvis() { beltSize = CURRENT_YEAR - 1930; }
  public static void main(String[] args) {
    System.out.println("Elvis wears size " + INSTANCE.beltSize() + " belt.");
  }
}

Elvis wears size 0 belt? Elvis wears size 73 belt? Elvis wears size -1930 belt? None of the above?

The answer is -1930 because of a recursive initialization in the initialization of INSTANCE. In the constructor, the value of CURRENT_YEAR is actually 0, because it was not yet initialized. To fix it, you need to move the initialization of INSTANCE after the initialization of CURRENT_YEAR. Moral: watch out for circular initialization.

Next problem is "What's the point":

class Point {
  protected final int x, y;
  private final String name; // Cached at construction time
  protected String makeName() { return "[" + x + ", " + y + "]"; }
  public final String toString() { return name; }
  Point(int x, int y) {
    this.x = x; this.y = y;
    this.name = makeName();
  }
}

public class ColorPoint extends Point {
  private final String color ;
  protected String makeName() { return super.makeName() + ":" + color; }
  ColorPoint(int x, int y, String color) {
    super(x, y);
    this.color = color;
  }

  main() {
    System.out.println(new ColorPoint(4, 2, "purple"));
  }
}

The output is [4, 2]: null because the subclass method executes before subclass constructor body. To fix it use a lazy initialization, by removing the this.name assignment in Point and introducing a synchronized String toString() in Point, that check the value of name ivar and assigns to it the first time. Moral: never call overridable methods from constructors.

Next is "Shifty":

public class Shifty {
  public static void main(String[] args) {
    int distance = 0;
    while ((-1 << distance) != 0)
      distance++;
    System.out.println(distance);
  }
}

31? 32? 33? None of the above?

None of the above: infinite loop because (-1 << 32) == -1. Moral shift distances are computed mod 32 or 64. It's impossible to shift out an entire int using any shift operator.

Next is "Line Printer":

public class LinePrinter {
  public static void main(String[] args) {
    char c = 0x000A; // \u000A is Unicode for newline
    System.out.println(c);
  }
}

Two blank lines? 10? Won't compile? It varies?

The character \u000A will be broken by the compiler in two, essentially making the comment line break at the Unicode word. Moral: Unicode escapes are dangerous, always use escape sequences.

JavaOne - MacOS X Java session

10:55am I'm now in the MacOS X Java Technology Overview session - The Ultimate Java Development Plan - presented by Alan Samuel, a "Java Technologies Evangelist" at Apple. Pretty good attendance, the room although small, is packed (more than 100 people I think).

11:02am It started. Most of the people in the room develop in Java on MacOS X, a lot more since few years ago, when the presenter asked the same question and got only 4-5 people doing Java development on MacOS X. Lots of talk about why use Apple hardware and software and develop Java on it. Tools: Apple's ProjectBuilder as well as third-party tools. Enterprise applications: Oracle 11i, SAP etc. Server strategy: WebObjects, Tomcat, 3rd party tools and J2EE servers running on Xserve. JDK 1.4.1 had over 1 million downloads to date.

In 1.3.1 there were 1842 Java classes and 900 custom Apple classes. In 1.4.1 there are 2991 Java classes and only 300 custom Apple classes. The AWT is re-implemented in Cocoa (rotated tabs, translucent windows). The JDK plug-in is now supported in Safari. MacOS X supports both JVM versions at the same time, with 1.3.1 being the default one. The JVM could be specified on a per-application basis.

JDK is optimized for MacOS X: Aqua look-and-feel, access to Quartz Extreme using just the standard Java APIs. The result is more responsive UIs, live re-sizing, smoother scrolling and snappy feedback. The Java Internet Plug-in has faster loading of applets, enhanced applet caching, integrated with Keychain and printing. Java WebStart updated to version 1.2, works with Keychain and adds automatic app creation. Keychain implemented using Secure Transport (?), certificates stored in Keychain. Good accessibility support, no need for Java Access Bridge: full screen reader support, full keyboard access and visual notification (hm, I wonder why the Tab key doesn't work to switch between most controls, including buttons). Support for handwriting support using Inkwell and for sound input. The new JDK version includes AppleScript support: scriptable UIs widgets. Shared system jars with system objects being shared between each VM (I guess very similar to shared libraries in the C world). This saves 3-6Mb per application with savings mainly in Swing and AWT-based apps - even more optimizations coming.

11:32am He's now giving a demo using Marratech Pro 3.4, an impressive Java collaborative tool that integrates video and a board for sharing documents. The software is built by a Swedish company. The presenter is Serge Lachapelle and he's talking from Sweden together with two other people, Magnus, Rolf and Rickard also online. The communication protocol is encrypted using 128bit keys. The application ran initially only on expensive Solaris boxes, but was easy to port to MacOS X on Java 1.4.1, including the native libraries written in C. (BTW, the sound quality is great, even though the video stream is pretty crapy, only 1-2fps).

11:42am Back to the presentation. Few tips on how to write Java apps for MacOS X, all of them documented on the ADC Web site. Don't use modifier keys directly, use Toolkit.getMenuShortcutKeyMask(). Not all look and feels are the same: use layout managers instead of explicitly placing the components. Query UIManager, toolkit for colors and icons.

Few words on Panther: we'll be "shocked" by the new operating system. It will be announced two weeks from now at WWDC at Moscone Center in San Francisco (the same location as JavaOne).

JavaOne day

Cold this morning in San Francisco, I almost regret I left home in just a T-shirt. I carpooled with Costin and the traffic was really good, no delays anywhere on the highway or getting into the city.

This year the number of attendees seems significantly smaller than that of the past years. The corridors are not as crowded as they used to be. The exhibit area has fewer exhibitors and smaller booths. Sun has the biggest booth; Borland, Motorola and few others have slightly larger booths that the rest of the companies. Lots of small companies occupy the rest of the smaller booths.

I arrived late for this morning's keynote. The part I caught was really boring, with a presentation of a RAD tool for Java. I can't believe people still do live coding during such shows. The developer using the tool had some minor problems with the code, making the presentation stale for a while. At this point I went out looking for wireless connectivity. I couldn't find any, but I did spot a place where I can hook up with Powerbook to a wired Ethernet.

JavaOne

Tomorrow I'll be at JavaOne, I'll probably be there for one day only, as I have a lot of work to do at office. I'll attend few sessions on Java in the Web Tier

• design patterns for high-load Web sites (TS-3588)
• StAX for efficient parsing of XML (TS-1195)
• an always fun session on Java programming puzzles (TS-1496)
• Per Bothner's session on Kawa (TS-2907)

Last year the wireless connectivity was pretty bad, with few base stations available and those were really overloaded by the number of attendees trying to read email, browse the Web, trade stocks, whatever. I hope this year is going to be better, otherwise I'll have to resort to using Starbucks' wireless connection (fortunately there are two shops right in the area).

If you're in the area and want to meet me, send me an email SMS at javaone@webweavertech.com, I will read email SMS at this address on my cell phone. Send me your cell phone number and I'll call you back.

Cocoon flow controller implemented in Java

Alex Krut has an implementation of Cocoon's control flow controller using ATCT - Asynchronous Transfer of Control Threading - a proprietary technology from Velare Technologies that emulates the behavior of continuations in Java.

ATCT's enhanced Java becomes the third language used to implement Cocoon's control flow. The first one was Scheme through the use of SISC; because of the little interest, support for it is not provided anymore in Cocoon. The second language, and currently the main one used in Cocoon, is Christopher Oliver's extended version of Rhino JavaScript, that incorporates continuations support. The Cocoon integration of ATCT appears to be pretty straightforward, the new flow controller simply implements the API provided by Cocoon. Great job, Alex!

The ATCT framework appears to provide the functionality of continuations in pure Java. A similar approach has been provided in Java by Brakes, which instruments the Java bytecodes to be able to capture the state of the execution at any given point in time. I'm not familiar with the way ATCT works, but the Web site claims ATCT is "a patent pending, pure J2SE and J2EE Java framework", which is a dubious statement to me. Continuations have been around for more than three decades, so I really doubt there's anything to be patented here. Anyways, I'd be curious to know more about this technology and what is that the actual patent covers.

Processing XML configuration files using XSLT

Charles Miller decided to use Scheme to describe configuration files instead of XML.

I have a different approach. Since most of the applications I write these days end up processing XML anyways, I prefer writing XML configurations file. Since I really like XSLT's power and to avoid the proliferation of tools like commons-digester and others, I write a simple XSLT stylesheet to process my configuration files. Such a stylesheet would need to receive as parameter a context Java object which needs to be initialized according to the configuration file.

The object initialization could be written in simple XSLT templates. Using Charles' example, lets say we have a hypothetic add-module XML element with a module attribute. We'd like to have a MyClass instance that gets invoked with a addModule method and passed the string value of module. The following XSLT stylesheet would do it:

<xsl:stylesheet version="1.0"
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns:MyClass="java:some.package.MyClass">

  <!-- This is the MyClass instance to be initialized -->
  <xsl:param name="context"/>

  <xsl:template match="add-module">
    <xsl:value-of select="MyClass:addModule($context, @module)"/>
  </xsl:template>

</xsl:stylesheet>

An instance of MyClass would need to be passed as parameter to the XSLT stylesheet from the Java side of the world. This can be set using the Transformer.setParameter(String, Object) method.

XSLT comes for free in Java 1.4, so there is no memory overhead from using XSLT. Even so, I still prefer Saxon over Xalan 2.3.1 that is distributed in JDK 1.4.

J2EE's PetStore with Cocoon's control flow

Christopher Oliver came up with a really nice implementation of J2EE's PetStore application using Cocoon's control flow!

The application controller is written using a modified version of the Rhino JavaScript implementation, that has first-class continuations and tail-call function calls elimination. The application is located here; check out the petstore.js application controller that drives the application, and the PetStoreImp.js that implements the data objects.

The entry points in the application are either top level JavaScript functions, or simply restart a continuation saved previously using the sendPageAndWait function call. Simple, yet very powerful!

New weblogger: Costin of Tomcat fame

Costin Manolache of Apache Tomcat fame finally started blogging. Welcome Costin!

Free software Java platforms

Richard Stallman is interested whether XSLT-process, my XSLT processor and debugger Emacs interface for the Saxon and Xalan XSLT processors runs on top of a free software JDK.

As far as I know Kaffe is one such free-software implementation of a Java virtual machine. There are many other implementations listed at Kaffe's Web site.

Does anybody have any experience with any of these in a real project? How much of the JVM functionality and the associated class libraries in JDK 1.3 are implemented? Can one run an XML parser and an XSLT processor like Saxon or Xalan on top of them?

XSLT-process 2.2 for Emacs released

I'm happy to announce a new release of the XSLT-process mode for Emacs. XSLT-process is an Emacs mode for applying and debugging XSLT stylesheets from within XEmacs or GNU Emacs. You can also use this mode to do processing of DocBook documents edited with Emacs.

Tony Addyman contributed most of the changes in this release. He's using Emacs and XSLT-process to teach XSLT to students in UK!

Here are the main changes since the 2.1 release:

• XSLT-process now uses Saxon 6.5.2, Xalan 2.4.1 and FOP 0.20.4
• Debugging using Xalan is now fully supported.
• Both JDK 1.3 and 1.4 are supported.
• During debugging sessions the output produced by executing the stylesheet is serialized immediately, then displayed and incrementally updated in an Emacs window.
• Added parameter passing to the XSLT processor. These are parameters that are specified to all the stylesheets, in addition to any other parameters. They are specified through the customization menu.
• Logging output, error messages, etc. generated by the FOP processor are displayed in the *Messages* buffer. The level of logging can be customized.
• Improvements to the display of information during debugging.
• MacOS X is now a supported platform, in addition to Linux and Windows 2000.

More on Java macros

Continuing the thread on Java language extensibility, Carlos comments on a private email I sent to Joshua Bloch and Gilad Bracha, the spec leads of JSR 201, with Carlos, James and Ken CC-ed on the email, regarding the use of JSE-like macros added to Java.

He points out that we should break the problem in two parts. The first is what should be the syntax used to define macros, and the second what is the syntax the macros are allowed to extend.

Carlos also points out that macro usage allow for the introduction of new keywords to the language, which will soon end up in name conflicts. He argues that we need to introduce namespaces for all these new keywords.

For the first part I agree that we need to discuss the issues involved. Personally I think the way JSE defines macros is good, although it may be hard to comprehend at times, especially for non-Lisp programmers. It's definitely not going to be the usual easy to read Java code, since the reader has to change his/her typical Java procedural thinking. As for the language extensibility part, I think the current definition is good, I cannot think of more ways to extend it or of possible shortcomings.

When it comes to keyword namespaces, I think we should avoid introducing these, or at least we should avoid having to introduce them in the keyword names. IMO this is ugly and has the potential of introducing changes in the compiler (which should be avoided at all costs because of high costs/impossibility in backporting these). I haven't found this to be an issue while programming Scheme or Lisp, and I don't think this could be a real issue with Java.

Another good point raised by Carlos is that macros cannot address things like autoboxing, which is a new feature proposed by JSR 201. I definitely agree. I think the scope of JSR 201 should be reduced to extending the language syntax only in ways which cannot be addressed by macros, currently autoboxing and static imports. The rest of the extensions (enumerations and enhanced for loop) should be defined as macros part of a standard library of macros, addressed either by this JSR or a different one.

A last comment I have is that the macros implementation should be easily backported to older versions of Java, so we can easy their adoption and ease the migration to this new feature.

JSR 201 thoughts

As I pointed out in previous threads, I think the JSR 201 should broaden the scope of the proposed extensions. Instead of proposing just some minor syntactic sugars to the Java syntax, it should try to extend the language in such a way that allows unlimited extensibility of the syntax by means of macros.

The idea of macros comes from Lisp languages, which use this to successfully extend the semantic of the language. In Lisp the core runtime engine deals with very few abstractions, everything else is implemented through macros.

A very interesting macro system written for Java is Java Syntactic Extensions (JSE) from MIT, which allows the language to be extended by developers, with no need for a JSR. The scope of JSR 201 for example is already implemented in few lines of code in JSE and is completely portable. No need for compiler or VM changes to support them.

Many of the proposed extensions being talked on Java blogs these days, like functions (with or without closures), blocks and many other syntactic sugars can be easily implemented with a macro facility, as demonstrated by JSE.

Here are some examples taken from the JSE distribution:

• enums

    public enum PowerWarning BATTERY_NORMAL, BATTERY_LOW, BATTERY_CRITICAL;
  

• asserts, which are completely portable, no need to upgrade to JDK 1.4 to use them:

      public void yuk (boolean x) {
          assert(x, "hello");
      }
  

• foreach:

      static void tst (List c, List l) {
          forEach (Integer i in c, Float f in l)
              System.out.println("contents: " + i + " and " + f);
      }
  

• when:

      static void tst (boolean b) {
          when (b) { panic(); }
      }
  

JSE works by preprocessing the extended source code and generating pure Java source code which runs on any Java platform. There is no need for any compiler change or JVM change. Very beautifully done I think!

The beaty of this approach is that one can extend the syntax of the language to support C# annotations, functions , blocks or anything else which is needed using macros and the existing Java language.

Why GPL should not impede using JSE

I've looked at JSE more carefully and I believe GPL should not be an impediment in using it. The reason is that JSE is used as a preprocessor and none of its code is actually linked in the resulting binary. JSE simply transforms everything to pure Java code which should compile cleanly on any JVM.

Now the only question is whether the macros defined by developers are covered by GPL or not. To me it appears that if the macro is just a syntax object, and does not depend on any of the classes JSE uses, the code is clean and the macro could be released under any license. And most of the time developers would develop such macros without going into the guts of JSE.

Java hygienic macros

In a comment on my hygienic macros for Java idea, Ken Meltsner points out Java Syntactic Extensions, a macro system for Java. Great stuff, thanks for pointing it out!!

Check out in the slides how the forEach statement (slide 30) is implemented in the system: the implementation fits in only one slide! The slides also show as example an accessible keyword (slide 39) which, when used to declare an instance variable, automatically generates setter and getter methods.

One thing I liked about the system is that it's procedural, thus potentially easier to understand and master than the declarative syntax macros in Scheme. Even then however, it could still be a challenge to understand and master.

Such a macro system would be the only thing needed in Java, everything else could be implemented using it. This is the case in most implementations of Scheme, which have a very small core and everything else is implemented as macros.

Extending the Java language: hygienic macros

The idea of hygienic macros like in Scheme for language with "non-regular", Lisp-like syntax is something which had me thinking for a long time. The challenge is how to provide not only semantics to new language constructs through macros, but also syntactic extensions to the language. Macros are possible in Scheme because of its uniform syntax, the new language constructs introduced by macros are just conventions translated to lower level constructs using the same regular syntax.

In Scheme you don't need any special support in the parser, macros are built entirely within the language. This is possible because of the simple syntax (or lack of it I should say ;) In Java this is not possible, you need compiler support for this to happen.

One idea would be to implement the parser for the Java language in an extensible way, which could allow new syntax extensions to be added at runtime. The only thing we would need to add new to the language is the ability to define "hygienic macros", e.g. ways to extend the syntax of the language understood by the compiler. The compiler's parser would need to be able to read these macros and record their definitions. In the process of doing this it has a chance to verify the new syntax described in the macro is still a valid grammar. During the syntax tree construction, it could invoke the defined macros to do tree transformation, which would transform the extended language into a pure Java syntax tree.

These Java macros would do nothing else than manipulate the syntax tree produced by the parser. Now while this would not be as powerful as Scheme's hygienic macros, it would still be a good alternative for all these language extensions. We won't need JSRs to extend the language in different ways, the only needed is a JSR to define the Java hygienic macro.

I wouldn't expect all the developers to start extending the Java syntax in their own way. Only experienced developers will do it. We could quickly have a library of such useful extensions which become used a lot.

New Java language features (JSR 201)

I dont't think we actually need a full compiler for the extended language to translate to the 1.2 and 1.3 JVMs. Since most of these changes (except enu ms) are only syntactic sugars to the language, thus maintaining the semantics of the language, a source translator could do the job. Unfortunately the source translator would have to do quite a bit of type checking to generate correct code, which makes it a non-trivial piece of code.

I'm sure there must be some tools out there that already do Java parsing, syntax tree construction and minimal type checking. Perhaps we can reuse thos e to create such a source translator. Once the source is translated to plain Java, it can be compiled by any 1.2 and 1.3 compiler. Ant or Maven could automate this process and make it really painless.

javablogs.com

If you have a Java blog, don't forget to register it at the awesome javablogs.com. It's a simple and effective way of keeping track of people working with Java. Kudos to Mike Cannon-Brookes for coming up with the idea!

Bean Scripting Framework finally at Apache

Chuck Murcko wrote me to say that BSF is about to finally transition from IBM to Apache Jakarta! The mailing lists are up, although no Web archives yet; to register yourself go to the mailing list Web page. A new 2.3 release of BSF should become available once the Jakarta BSF Web site comes up, which should be up any time now.

I'm working on an MVC Web application framework which uses scripting languages supported by BSF as an option to write the Controller. BSF is a central piece in it, that's why I'm so keen on seeing it healthy at Apache. I am also planning to use some AOP patterns to provide extensibility to this framework. More on this as code becomes available.

JRuby

Interesting stuff, especially JRuby. As it has a BSF interface, Ruby becomes one more language you could script your Java applications in.

Java SSH implementation

I don't remember where I found this, I'm sure it was on a Weblog I'm subscribed to, but I cannot find it anymore. JSCH is a pure Java SSH2 implementation released under LGPL.

This should allow things like connecting to remote Unix servers from a Java application. Until now this was possible only by exec'ing the ssh command line tools. I'm not sure though JSch supports authentication through public keys though, which is very convenient as you don't have to ask users for the server password.

O/R mapping: Hibernate

For the last week I've been playing with Hibernate, an object/relational database mapping tool.

The documentation is good, you get the basic ideas very easily. There are some things which are not described in the docs, but since the sources and good examples are available, you can figure these things out fairly quickly.

I went with the top-down model to define my application. I defined the Java model first, and had Hibernate generate the initial mapping. I then modified this mapping to match my Java model more closely, and now I have a working system.

The application I want to build is a user feedback system for documentation. I plan to extend it to use it in a user feedback system for a Weblog tool as well. Fully integrated with the Blogger API and email too, so readers and people that comment can get notifications by email when somebody posts a new comment. I felt this need in my blog, and I'm sure other bloggers felt it too.

Object/Relational bridges

Gerhard Froehlich writes:

This OR/Bridge thread is amazing and it's was me who kicked it off. I learned so much last days. Blogs are really great. Lot of smart people around who have the same problems every day and I think we get them all solved.

I feel the same here, there are many new things I've learned by following this thread. Next thing to do: find out more about Hybernate, and what is the difference between it and Apache OJB.

JPetStore

Christopher Oliver points out JPetStore, a reimplementation of the classic J2EE PetStore application. Chris is looking at replacing the front-end JSP/Struts implementation of JPetStore with a Cocoon based implementation that uses the control flow engine.