Wednesday, 28 November 2012

Improving the MapGuide API Documentation (part 3)

This post has a bit of a misleading title (it's more the wrapper API we're improving than the documentation) but whatever, we'll run with it.

So just to explain the previous screenshot (and probably my java-related tirades on Twitter and Google Plus for anyone who happens to follow me there) ...

That screenshot was "visual" confirmation of some MapGuide API wrapper enhancement work I've been doing to address some of the pain points of using the MapGuide API (namely Java, but this work has some cross-pollinating benefits for .net as well)

The Problem

For the 15 people in the room who actually use the MapGuide Java wrapper API in its current form (I kid, though it could possibly be true), you've probably had to deal with these annoying issues:
  • The Java proxy of MgException being a checked exception. Thus having to pollute your java code with "throws MgException" in calling methods or having to try/catch an exception that probably isn't thrown 99% of the time and if you do catch it, you're probably just gonna log or display that exception anyway.
  • Java class method naming being a verbatim transplant of their C++ counterparts (ie. They're in UpperCamelCase instead of lowerCamelCase)
  • Java proxies of MapGuide collection classes being verbatim wrapper classes, and not behaving like a Java collection. You can't even for-each through these collections due to key java interfaces not being implemented in the proxy classes.
  • Needing to have the web-based API reference on hand as doxygen commentary is lost in translation to the target language. Auto-complete in your Java IDE of choice gives you no documentation, because none is transferred across.
  • Needing to have the web-based API reference on hand to also find out if a given class or method is deprecated, because nothing in the proxy classes will tell you that to trigger any compiler warnings
  • Hardcore memory leaks in the Java wrapper
Most of the above points, pretty much summed up in this 6-year old ticket.

Though the memory leaks were plugged for the 2.4 release, the other problems are still present. Despite my own reservations about Java (the language), Java (the platform/ecosystem) can't be ignored. Just because we're using SWIG to churn out bindings to 3 different languages doesn't mean we can't do some tweaking here and there for a given language. We already do this for .net (to support properties and serializable exceptions), so why not give the Java wrapper some love as well?

The Solution

For those who are wondering how we actually build bindings to the MapGuide API in 3 different languages, here's a high-level overview:

As already mentioned, SWIG is the tool used to generate the language bindings to the MapGuide API, but in order to enforce some level of encapsulation in the wrapper APIs (because there's C++ internals in some MapGuide API classes that shouldn't be exposed in the wrapper classes), we run a custom pre-processor (IMake.exe) through the MapGuide C++ headers to generate a "sanitized" input file for SWIG to do its thing.

If you ever look at a any MapGuide C++ header and wonder what those PUBLISHED_API, INTERNAL_API and EXTERNAL_API markers are for, they are for IMake to generate an encapsulated view of that particular class for SWIG. Similarly, the __get and __set markers you see at the end of some method declarations are hints to IMake to generate .net properties for that particular method.

The same IMake tool is also used to generate the constants.php for PHP and the necessary constants source files for .net and Java

The solution consists of 3 parts:
  • Modifying SWIG to support our java-specific requirements
  • (Ab)using some SWIG directives for purposes beyond their original scope
  • Modifying the IMake preparation tool to take advantage of these (ab)used SWIG directives
Modifying SWIG

This part was simpler than I thought. The SWIG source code was not as hostile as I originally thought and hacking the java module to support what we need was relatively straightforward. SWIG was basically modified to support 2 optional java-only flags:
  • To omit the "throws MgException" clause in any java method generated
  • To output java methods in lowerCamelCase. Since all MapGuide API C++ methods are guaranteed to be in UpperCamelCase, this is a simple case of lower-casing the first letter of each method that SWIG writes out.
These flags are optional, because if we are going to provide a new enhanced Java wrapper API, we'd still want to keep the existing one around for compatibility purposes, with the enhanced wrapper being an opt-in choice.

(Ab)using SWIG directives

The directives in question are:
  • %typemap(javaclassmodifiers) (.net equivalent is %typemap(csclassmodifiers))
  • %javamethodmodifiers (.net equivalent is %csclassmodifiers)
Normally, these 2 directives are used to alter class/method visibility of the respective generated proxy classes

For example, if we apply this SWIG directive:

 %javaclassmodifiers MgMap::MgMap() "protected"  

We get the following java proxy class method declaration:

 public class MgMap
   protected MgMap()  

Similarly if we apply this typemap:

 %typemap(javaclassmodifiers) MgMap "private"  

We get the following java proxy class declaration:

 private class MgMap  

The trick with these directives is that that SWIG does no validation of the visibility string, meaning we can put any arbitrary content in there, say things like javadoc comments and annotations (or XML comments and attributes in .net). In fact, this technique is what the authors of SWIG recommend as a way of being able to document your proxy classes

So since we all know that the blank MgMap constructor is a deprecated API that you should stay away from. We can specify SWIG directives like this:

 %javamethodmodifiers MgMap::MgMap() %{/**  
  * This constructor is deprecated. Use MgMap(MgSiteConnection) instead  
 @Deprecated public%}  
 %csmethodmodifiers MgMap::MgMap() %{///<summary>  
 /// This constructor is deprecated. Use MgMap(MgSiteConnection) instead  
 // </summary>  
 [Obsolete(\"This API is deprecated\")] public%}  

Which will then turn into the following proxy class method declarations when SWIG processes them:

  public class MgMap   
   * This constructor is deprecated. Use MgMap(MgSiteConnection) instead  
   @Deprecated public MgMap() {   
  public class MgMap   
   /// This constructor is deprecated. Use MgMap(MgSiteConnection) instead  
   [Obsolete("This API is deprecated")] public MgMap() {   

Meaning with these directives, it is now possible for us to not only pass down (converted) doxygen comments, but also pass down deprecation intent as well, because sometimes an API reference is not enough to tell you something is deprecated. You ideally want csc or javac to throw a compiler warning at your face while compiling against the wrapper API library to truly drive the point home that you should not be using this class or method.

Modifying IMake

So we have the means to transfer documentation and deprecation from the C++ classes, but surely we aren't going to manually transcribe the documentation fragments of the hundreds of classes that are in the MapGuide API? Of course not!

The IMake tool already does a pass through all the MapGuide C++ headers. And if you look at the constants.php generated by IMake, you can see that it does pick up and collect doxygen commentary along the way (albeit dumped verbatim into the target language, making it un-usable as javadoc or .net documentation). So the IMake tool was modified to do the following:
  • Translate these collected doxygen fragments to their javadoc or .net counterparts
  • Write these fragments out as the aforementioned SWIG directives for the target language
Other assorted codegen magic

Sadly, Java being the dinosaur of a programming language that it is compared to C# (in my opinion), does not support something similar to partial classes, which would've made augmentation of generated java proxy classes so much simpler, so in order to add java.util.Collection and java.util.Iterable support, we have to once again look at SWIG directives for ways we can augment the generated java classes.

Fortunately SWIG also supports a javacode typemap, that basically allows us to inject vast bodies of java code into our SWIG generated java proxy classes. This allows us to easily implement java.util.Collection support for the collection classes and java.lang.Iterable support allowing for these objects to be iterated in a for-each fashion.

In a way, these SWIG directives are kind of like pseudo-templates for Java, minus the pain of having to deal with the half-assed Java generics, as their way of implementing generics completely blows (I can't make an array of generic type T? GTFO!)

The MgException class (from which all MapGuide exceptions inherit from) extends a pre-defined java AppThrowable class, which itself extends from Exception. As a result, all proxy exceptions are checked as a result. Making these exception classes un-checked as a simple case of re-basing the AppThrowable class to extend from RuntimeException instead.

The End Result

So how well do these changes fare? We can verify by running javadoc against the generated proxy Java class source files. Have another look at that same screenshot of the javadoc from our "enhanced" Java wrapper API:

What can we see here?
  • Java method names are now properly named in lowerCamelCase
  • Doxygen \deprecated markers have been properly translated to @Deprecated annotations (and ObsoleteAttribute in .net)
And also a simple reminder from IMake that we've still got lots of room for improvement in the API documentation content

If we dive down to the method level, here's what we currently have in doxygen:

And here's what gets transplanted to javadoc

As you can see, most of the key doxygen directives have matching/compatible javadoc equivalents allowing for easy conversion by the IMake tool.

All MapGuide exceptions are now un-checked, as they extend RuntimeException

For .net, we get the same result as going through the DoxyTransform tool. In fact, this API enhancement work will render the existing .net documentation approach obsolete, as this approach does the same thing but supports multiple target languages.

So the javadoc output is proof that we have working transplanted documentation, now comes the important bit: making this documentation easily consumable from within a Java IDE, which is incidentally where I'm currently scratching my head. 

What is the official/recommended way to consume Javadoc in a Java IDE (Eclipse, Netbeans, etc)? Is there an official way? Or does each IDE have its own way? I'd appreciate some pointers from someone more knowledgeable in Java tooling than myself. Feel free to comment below.

At the moment, from what I can gather there's 2 ways to bundle up this documentation for consumption within a Java IDE:
  • Pack the java proxy class source files into a separate -sources jar file
  • Pack the javadoc documentation files into a separate -javadoc jar file 
I tried both approaches through IntelliJ IDEA and it seems to pick up both forms:

Here's the result of ctrl+Q with the -sources jar

And the result of ctrl+Q with the -javadoc jar

But I'd still prefer to only have to choose one method that will work on all the major Java IDEs. I'm just not sure which one it is. I'm currently leaning on the -source version, but I'd still like an "official" answer.

But what about PHP?

You might have noticed that PHP is a conspicuous absence from this MapGuide API enhancement work.

Well the truth is that PHP poses the following problems:
  • Besides constants.php there is no actual PHP source files to "compile" into. So there's no actual place to drop our translated doxygen comments into.
  • There is no officially defined standard for documenting PHP code. Even if there was, the real aim of this API enhancement work is to improve integrated API documentation (ie. From within an IDE). If all the PHP documentation solutions ultimately produce something similar to what we already have in our API reference (a web-based documentation source), then what is the real point? We already have that!
What's left?

Besides answering the Java IDE documentation question, this API enhancement will need a requisite test run on Linux (especially the SWIG/IMake stuff). All the viewers still work as before on Windows post-SWIG/IMake surgery, indicating the wrapper APIs haven't been negatively affected in any way.

Also I have to draft up an RFC for these enhancements, for eventual inclusion into the next major release of MapGuide Open Source (2.5). If everything pans out, this next release will have 4 Web API options:
  • PHP
  • .net (with XML documentation files for Visual Studio)
  • Java (the old crufty one, kept for compatibility purposes) with Java IDE documentation support (whatever it will be)
  • Java (the enhanced one with the changes described here) with Java IDE documentation support (whatever it will be)

Monday, 26 November 2012

Improving the MapGuide API documentation (part 2.5)

This is the latest result of some SWIG/IMake sandbox work. So far, so good!

Tuesday, 20 November 2012

Announcing: MapGuide Maestro 5.0 beta 5 and 4.0.4 maintenance

Here's another beta for Maestro 5.0 and a matching 4.0.4 maintenance release.

Since 4.0.4 is a maintenance release there isn't much to talk about other than there's bug fixes :)

So let's talk about the 5.0 beta instead. Here's what's new.

More SharpDevelop goodness

As already mentioned, we've expanded our use of SharpDevelop code and components to give us improved versions of existing functionality:

  • A smarter, more responsive IronPython REPL
  • XML folding support for the Generic XML editor
  • FDO Expression Editor syntax highlighting
Sadly, Mono did not want to cooperate with us despite our code (and the SharpDevelop version we're using) sticking to the Mono compatible subset of the Base Class Library + Windows Forms. So the zip distribution (which you would install for Mono) does not include the IronPython REPL functionality and uses the existing FDO Expression Editor implementation on Mono.

Local resource previews

As part of eating our own dogfood, previews for certain resources will now use our own built-in map viewer component. The main benefits of using a locally-based previewing mechanism that:
  • It is generally snappier as we don't have to spin up an external web browser to show the preview
  • It is smarter about the initial view if we're previewing layers where the scale range is not 0 - Infinity
  • Our preview UI has more niceties over the browser-based previewing mechanism
If this feature is not working for you or you prefer the old way of previewing things, you can turn this option off in the Application preferences.

Updated SDK

This is the first release in the Maestro 5.0 series to include a matching SDK. This includes some new samples
All these samples have been migrated to .net Framework 4.0 / VS 2010 (You can use the VS 2012 express editions as well if you so choose). You cannot use the Maestro 5.0 API in a version of the .net Framework that is older than 4.0.

As I hinted to when I started on the road to Maestro 5.0, the new SDK also contains a build tool to "roll your own" LocalNative provider, which lets you use the MaestroAPI over the official MapGuide API. This is a much more convenient way than me having to build a LocalNative provider for n specific versions of MapGuide

Simply point to where your official MapGuide .net API assemblies are and hit Build to build your LocalNative provider. Then you can use this provider by registering the dll in the ConnectionProviders.xml of your MaestroAPI application.

Please Note: Because this is still a beta, the Maestro APIs are not set in stone. APIs may still change from now until the final release.

Other cool stuff

We've tweaked the WMS Configuration UI to better handle some of the more obscure WMS servers I've encountered in my journey thus far.
  • You can now edit the FDO logical class name/description
  • You can also swap the class name/description individually or in a batch. This is a convenience function for some WMS Servers where the layer names actually end as the FDO logical class description instead of class name when processed by the WMS FDO provider.

We've also made the FDO class name encoding/decoding in Maestro to behave more like its FDO counterpart so that problems like this should no longer happen anymore. We've backed this up with the same test suite and data used to exercise the name encoding/decoding in FDO proper. Basically, you shouldn't get strange looking FDO class names anymore. They should get properly encoded and decoded.

When deleting a resource with dependent resources, we're now nice enough to tell you what they are

For folders, we just give you a catch-all question which may or may not be true, simply because we don't want to waste CPU cycles computing affected references if we happen to be trashing a big folder.

There is also a new context menu option for selected resources that allows you to compile a full dependency list for a given resource

This produces a list of resources that the selected resource either directly or indirectly depends on, which can be saved to a text file

This feature combines nicely with an improvement to the Package builder to allow packaging an explicit list of resource ids instead of a particular folder. Because sometimes, you just want to package a specific resource and everything it depends on, instead of a particular folder which may then package a whole bunch of other junk which you don't particularly want.

On the theming front, ColorBrewer color ramps can now be reversed if you so desire.

We've improved your ability to rapidly translate Web Layout and Flexible Layout documents, but previewing such documents (or any browser-based preview actually) in the AJAX/Fusion viewer will still default to using the en locale. You can now choose what locale that browser-based previews will default to in the Application preferences.

This has the effect of basically tacking a locale=your_selected_locale to whatever preview URLs you launch from Maestro

So with that all said and done ... Go forth and download!

Monday, 19 November 2012

FDO Toolbox 1.1

Here's a new release of FDO Toolbox.

The new items are:

Please ensure you have .net Framework 4.0 and the Visual C++ 2010 redistributable (32bit or 64bit) installed before installing FDO Toolbox 1.1

Note: FDO Toolbox is still a semi-retired project. Don't expect any major new features. I've put this release out because I believe every new version of FDO should at least have a matching version of FDO Toolbox that uses that particular version.

Thursday, 15 November 2012

A work in progress

I wonder if the favicon might give you a hint? :)

Monday, 5 November 2012

Eating your own dogfood

For those who haven't heard the term, eating your own dog food as defined in Wikipedia:
Eating your own dog food, also called dogfooding, is a slang term used to reference a scenario in which a company (usually, a software company) uses its own product to demonstrate the quality and capabilities of the product
For the next beta of Maestro, we're doing some dogfooding of our own. The new Map Viewer component, will now be the default method for previewing the following types of resources across all connection types
  • Layer Definitions
  • Map Definitions
  • Watermark Definitions
Where previously, a web browser would be launched (if using a http connection) or a mg-desktop based viewer would be loaded (if using a local connection), for such resources we have our own preview UI built on top of the Maestro Map Viewer component (which also works in Mono)

The main benefit of this method of previewing becomes apparent when you try to preview layers with restricted scale ranges. The default method (launching the AJAX viewer) for such layers will usually result in seeing nothing as we don't have the ability to pass on the initial opening scale to the AJAX viewer nor do we have the required information to adjust the preview bounds. That's why as a workaround, we put in a "Zoom to Scale" command under the "Tools" menu of the preview Web Layout to let you easily zoom into the layer's visible scale range.

With the new preview method, the preview map viewer will automatically adjust its initial scale to fit into the layer's scale range, eliminating the need to manually zoom into the layer's visible scale range.

This feature is optional, and if you still prefer the old way of previewing resources, you can simply uncheck the option.

Next beta is almost here. There's some Mono-related bugs/annoyances to fix up