Matt and I have been experimenting with part of the JNLP API called SingleInstanceService.  Basically, it's a sort of super-static singleton class that's shared between all running instances of the application.  This is really cool in theory, as it allows you do do things like count the number of running instances, and allow only one.  There's some great code implementation examples [1], as well.

Of course, like all things, it can end up complicating things in practice.  Matt ran into what appears to have been a jar signing error caused by the addition of the SIS code, and was getting stuck.  I'm sure there's a reasonable solution to the issue, but before spending too much time hunting it down, we revisited the initial problem, which was to prevent the launching of a second instance of the application.

The best way to do this in custom code, would probably be to open up a socket connection on an unused port, and send a kill message, then open up a listener on that same port and monitor for one.  This would probably work fine (assuming there's not already another application using that port), but is there an even simpler solution?

After deciding this was more of a user convenience feature than a mission-critical failsafe, we decided to go the simplest route: put a token file in the user directory.  Subsequent applications could check for this file, and if found, display a warning to the user to ask if they'd like to quit the application.  This allows a manual override, in case a prior instance never exited normally.