Monthly Archives: July 2012

Notes on using JSDuck for javascript API documentation

I’ve been using JSDuck to generate log4js-ext documentation, and I’m quite happy with it.

It allowed me to generate the log4js-ext API documentation in a day, and I had to learn how to use it and decide some baseline practices in that time.

These notes are relative to JSDuck 3.11.0, and include some practices, doubts and precautions I consider relevant.

BTW, the list of tags JSDuck supports is here.

Important practices -at least to me!

  • Data types: this is key, because javascript is not type safe and sometimes the type is not evident

    • Always specify the type for parameters, return values and properties.
      This is usually done writing {xxx} somewhere, where xxx is the type (String, Array, Sm.log.Logger,etc.).
    • If something can have several types, use this notation: {String/Sm.log.Logger}, with ‘/’ separating several alternative types.
    • If you want to refer to the special arguments variable, you can define its type as {Arguments}.
    • Always add @returns {void} to functions that returns nothing. Else, you never know whether there is no type because the function returns nothing, or you forgot to specify it.

  • Statics:

    • You must always add the @static tag by hand, even though you are commenting something in statics: else, the static member will appear as an instance member.
    • You must specify @property explicitly for statics, or they will appear as a global, instead of appearing as part of a class. Bug?

  • If you are using the config thing in ExtJs based code:

    • You must explicitly add the @cfg tag, or else the item will be signaled as a property, not a config.
    • To get getter/setter doc you must add the @accessor explicitly.
      Unfortunately, if a config entry is not marked with @cfg, there is no documentation generated for getters/setters.
    • If you have an item for which you don’t want a setter, create documentation for the setter and make it private.
      As explained below, you can create documentation for an item even if you don’t declare it explicitly.
    • Mark the config as required if needed: you do this by writing something like @cfg (required)

  • Always add the @readonly tag where appropriate. It is very important because this tells you not to juggle with something.

  • Always specify the @protected and @private tags. Because in javascript everything is public, your only way to know about this is the documentation.

  • Always specify the @abstract tag where appropriate. This will help implementors of derived clases quite a lot.

  • In a @returns tag, do not start comment in one line and continue in another, or it will not be considered part of the return comment. You can start the comment in the second line, placing the type in the first one.

  • To hide a class to documenation, make it @private.

  • In javascript you can always add a new property or function as you see fit, but, how do you document something you add in the middle of some function?
    You can document things that are not declared explicitly like this:

         * @cfg {String} name (required)
         * A very importand config option that must be specified.

    I have to confess that it feels a bit strange to have comments for a property that is not there, and that I feel better if I provide the property itself, if at all possible. But I have to consider what’s worse, a ‘lonely’ comment or a comment with a property initialized to a fake and posibly wrong value.


There are some things I’m not sure how to handle:

  • I’m not sure about how to specify an enumerated type (WARNING, INFO, …).
    Maybe the best way is to create a class just for that enumeration, and refer to that class as the type.
    I think this is good style, too, so everything should be ok with this approach.
  • I’m not sure about how to specify optional arguments: for example, in log4js-ext a logger’s logging methods can receive from 1 to n arguments, with very different purpose. I have not decided on a way to document that, beyond providing a link to examples.
  • I miss an additional visibility qualifier, to designate those methods visible only inside a subsystem (package, library, a group of classes…).
    Right now I just mark these as private, and add an extra ‘Package’ comment there (which is not visible in the API documentation, unfortunately).
  • I’m torn about when to make something private, and when not to document it.
    This being javascript, I feel that documenting a private method to the public is almost like telling them that they might use it if they are careful enough, and that to make it really private I should just avoid documenting it.


And there are some things that make me wonder if I’m missing something…

  • The @template tag, does it really make sense to use it? Its meaning, to me, is equivalent to implying that a certain function is overridable, but, is not that the case for functions,
    by default?

Checking everything is ok

Once you’ve generated API doc, you’ll want to:

  • Take a look at warnings -though JSDuck generates a lot.
  • Look for the ‘unknown type’ string in JSDuck output!
  • Take a look at the API doc page with all your classes, and look for spurious globals.
  • In some cases, I’ve got warnings about missing images…that were there.
    When checking warnings, make sure that they are for real.
  • One of these days I will create doc that includes the ExtJs doc itself, so that the user can check whatever he wants *and* I do not get a thousand ‘class not found’ messages when generating the doc.

Log4js-ext 1.0.1 is out!

Log4js-ext 1.0.1 is out!

This version just adds API documentation, generated via JSDuck, and a fix for IE 8. The documentation looks like this:

Log4js-ext API documentation

I have to say that I’m quite happy with JSDuck as a doc generation tool for javascript!

Unfortunately, the documentation is not inline due to googlecode hosting limitations: to take a look at it, open index.hml in the installation directory and click the API documentation link.

Version 1.0.1 provides a fix for IE 8, too, so that log4js-ext works ok when the development tools are not active.

Juicer: issues and workarounds

Juicer is a nice tool to merge and minify several CSS files into just one file. Kudos to the guy that created it!

That said, I’ve been bitten several times by it. Here are some of the troubles I experienced and what I did to get things to work for me.

Juicer: embedding images in CSS

Juicer wants to know the –-document-root when embedding images with --force-image-embed or --embed-images data_uri.

Note that I’m talking about image inclusion in a css using relative URLs, I have not checked this for absolute urls because I don’t care: I never use then.

For things to work, the --document-root must be the directory of the css file you are processing.

Again, a clarification is in order: I always process just one css file each time with Juicer, because I make a point of creating a single css file that @imports all css files I want to consolidate. The net effect is that I’m processing several files, but specifying just one in the command line.

I’m making this clear so that you know I just haven’t tested Juicer behavior with --document-root and multiple css files in the command line. Better safe than sorry!

If you notice I’m being defensive, it is because I am. Juicer is great, but I’ve found its ways are not obvious to me and made many assumptions that did not hold.

Fixing relative CSS urls with ant

I run juicer from my project ant tasks, and that can be problematic because there are ant ways and there are Juicer ways, and they collide from time to time.

As point in case, I have to perform text substitution on the generated css file so that relative urls are right: this is due to the fact that Juicer seems to be intended to be run from the webapp root directory if you want correct relative urls in your css.

*But* I have not managed to run the ant task in the webapp root directory. I tried the dir attribute in the exec task, that seemed to be the cure, but that didn’t do what I needed.

Yes, I can bypass ant and run things from the command line. But then I will have ant’s way, the command line way, and will need to duplicate configuration information for ant and batch files (quick, where is the YUI compressor jar in your system/project?). And I hate breaking the DRY (Don’t Repeat Yourself) rule.

Workaround: replace ../WebContent/ with ../, WebContent being the subdirectory relative to my ant build.xml file. This is way simple in ant:

   token="url(../../WebContent/" value="url(../"/>

Juicer is written in Ruby, and Ruby has its ways

Ruby seems not to like ‘:’ or ‘\’ in file names -or Juicer uses that in a bizarre way.

Ok, I know almost nil about Ruby, but I had some bizarre problems with file names. In some cases, the error message I’ve got helped me clearly diagnose the problem.

In other cases, I had no error message, and it took a *lot* of time to identify the problem.

The workaround is avoiding ‘:’ or ‘\’ in file names passed directly or indirectly to Juicer.

Note that this can happen inadvertently. For example, if you execute an ant task in Windows, ant’s ${basedir} can end up being c:\x\y\z.

Workaround: convert all directory and file names to Unix style, which Windows handles correctly, like this:

<pathconvert property="juicer-friendly-input-dir" targetos="unix">
  <path location="${basedir}/WebContent/css"/>

Juicer does not generate a non-minified css: but you can trick it into it

Juicer provides two nices features for CSS: consolidates several css files into one, and minifies the resulting file.

I would love to have a single production time css that is minified, and another one that is not, to help users of my js libraries with customizing the css. But Juicer does not provide a way to generate a non-minified version.

Workaround: specify a fake inexistent .jar file with the --path INEXISTENT_MINIFIER_JAR_TO_GET_NON_MINIFIED_CSS argument. Juicer will generate the consolidated css and will fail with a Unable to access jarfile error…but the consolidated css file with be there.

Yes, ugly, but very useful. I hope they had added a none option to the --minifyer argument that would have the same effect without the rror message, but they did not

Log4js-ext 1.0 is out!

Version 1.0 of log4js-ext, my logging library for ExtJs, is out!

It adds some extra functionality over 0.9 (which was almost a 1.0 candidate), but the main attraction is that it’s been polished to the point of being a product. You know: documentation, multiple file consolidation into a single file, minification, etc.

Here is the new features and improvements list:

  • Detail view for individual log entries:
    The built-in log viewer can show log details in an expanded view.
    That’s great for complex logged objects that will be happier formatted as multiple line JSON.
  • Documentation:
    The wiki provides information for users to be proficient with log4js-ext in no time.
  • Javascript and CSS consolidation and minification:
    This makes using log4js-ext easier as now all that’s needed is to add a single .js and a single .css to use all of log4js-ext.
    Besides, minification has reduced size to less than a third of the original size.
  • Refactored formatters to layouts:
    This is to follow log4j naming, a good thing because there is a thousand pages about log4j out there that might be useful in one way or other to log4js-ext users.
    Should have zero impact, as I doubt there is anybody out there creating custom formatters/layouts as of version 0.9

Besides, the library passes almost thirty unit tests in the latest Firefox, Chrome, Internet Explorer, Safari and even Opera versions.

Minificación + compresión = 90% de ahorro en espacio

Log4js-ext cuenta con una docena de clases javascript, algunos “mods” a clases de ejemplo o de terceros, media docena de archivos de tests para JsTestDriver (también javascript), y varios archivos .css, con lo que la librería se va a casi dos docenas de archivos.

Hace unos días decidí publicar la versión 0.9 de log4js-ext. Una vez que uno hace una librería pública empieza a pensar en la conveniencia de pulirla.

Parece importante consolidar tanto los archivos javascript como los .css en un par de archivos, para facilitar su uso. Y minificar esos archivos para optimizar el rendimiento.

Consolidar y minificar javascript

La verdad, ExtJs, librería sobre la que funciona log4js-ext, proporciona una herramienta para consolidar/minificar javascript, pero la documentación es floja, y me ha parecido relativamente complicada.

Entre otras cosas, hay que generar un archivo de configuración que debe listar explícitamente todos los archivos javascript, nada de ‘añádeme todos los *.js en el directorio xyz’.

El problema no es que esto sea incómodo, ¡que lo es!, sino que fácilmente termina por generar defectos: añades un archivo de nada con una docena de constantes y se te olvida incluirlo en el archivo de configuración de la herramienta. Siendo javascript como es de permisivo, ni lo notas hasta varios días más tarde, cuando ya la has liado.

O sea, que al final he pasado de la herramienta de la gente de ExtJs/Sencha y he usado ant más YUI compressor.

Porque a) ant me es familiar, b) ya lo uso en ese proyecto para otras tareas (cuantas menos herramientas, mejor) y c) ant tiene unos mecanismo de inclusión/exclusión de archivos muy potentes, que me permiten incluir en la versión minificada de la librería justo los .js que quiero, excluyendo los de test, ejemplos, etc.

Aquí va la parte de la tarea que genera tanto un .js para desarrollo (para depurar) como un .js minificado (para producción): el primero lo monto a base de concatenar todos los .js, y el segundo usando el YUI Compressor sobre dicho archivo.

<target name="generate.packed-files" 
               description="Generate a single minified .js + a single .css">
   <property name="pack.outputdir" value="${basedir}/workdir/packed-files"/>
   <property name="yui.jar" 

   <delete dir="${pack.outputdir}"/>
   <mkdir dir="${pack.outputdir}"/>

   <!-- Concatenate all .js files -->
   <concat destfile="${pack.outputdir}/${}-all-dev.js" 
     <fileset dir="${basedir}/WebContent/sm">
       <include name="**/*.js"/>
       <exclude name="**/*Test.js"/>
       <exclude name="**/test/*.js"/>
       <exclude name="**/examples/*.js"/>

     <fileset dir="${basedir}/WebContent/smthirdparty" >
       <include name="**/*.js"/>
       <exclude name="**/*Test.js"/>
       <exclude name="**/test/*.js"/>
       <exclude name="**/examples/*.js"/>
   <echo message=
         "Generated unique dev time file ${pack.outputdir}/${}-all-dev.js"/>

   <!-- Compress resulting .js file -->
   <!-- Use YUI minifier -->
   <exec executable="cmd">
     <arg line="/c java -jar ${yui.jar} ${pack.outputdir}/${}-all-dev.js -o ${pack.outputdir}/${}-all.js"/>
   <echo message="Generated unique minified js file ${pack.outputdir}/${}-all-dev.js"/>

Nótese que concatenar a saco funciona porque el código .js que escribo permite concatenar archivos .js sin problemas: esto no es una limitación impuesta para poder generar un archivo único (eso sería mala cosa), sino que sale de forma natural al aplicar una serie de buenas prácticas que me parece importante aplicar en todo el código javascript en modo “no hola mundo”.

Ejecutando este script genero tanto la versión de desarrollo de la librería (log4js-ext-dev.js) como la versión de producción (log4js-ext.js): la primera me ocupa 96.208 bytes, y la segunda 31.427 bytes.

Por cierto, usando Tomcat como servidor y activando la compresión gzip, el tamaño del archivo final enviado al navegador pasa a ser de unos 9.300 bytes, es decir, un ahorro de espacio con respecto al tamaño original del código de ¡más del 90%!

Consolidar y minificar los .css en un único archivo

Otra cuestión a resolver es la de consolidar los diferentes .css en un único archivo, cómo no también minificado, para lo que uso juicer , una herramienta escrita en Ruby -llamándolo desde ant. Los resultados son también espectaculares.

DirectJNgine 2.2 final is out

The new DirectJNgine 2.2 is out.

Finally I decided that extra exception information would made it in the final version, despite being experimental in the beta: the only change with respect to beta 1 is that we have decided to add a where property to all exceptions, making it possible to get the whole stack trace, but *only* in debug mode.

DJN 2.2 provides the following new features:

  • Automatic conversion to a single element array when calling Java functions that expect an array but receive a single value.

    This one helps get around the annoying behaviour ExtJs exhibits in some cases, as it can call a Java function with a single value or an array of values. Java, being type safe, can’t handle that.

    This means the following javascript code will work

    MyAction.myFunction( 'hi', function(...
    MyAction.myFunction( ['hi','bye'], function(...

    against the following java code,

    public void myFunction( String[] values );

    because DJN will turn the ‘hi’ sent by ExtJs for the first call into a single item array that will be acceptable to the Java function.

    Please, note that this auto-conversion to arrays works only for the parameters themselves, not for values nested in the received data.

    I feel that would be giving too much undue ‘flexibility’ to the client side: this auto-conversion trick is more of a workaround for ExtJs ‘liberal’ behaviour.

  • Added default Date serialiation, as explained here.

    The default way dates are serialized by Gson is not very useful or easy to handle, and I decided to provide a custom default implementation that is easier to work with.

    Just remember to check this link to make sure you understand how to handle it, and don’t forget to add the required code in the ExtJs side

  • Enhancements to unexpected server exception handling

    Right now, when there is an unexpected exception at the server, DJN provides a message, as per the Ext Direct specification.

    While this message provides both the exception type (without the package) and the exception message, I would like to have them apart. This way, it would be possible to handle exceptions without having to dissect the message, making things easier.

    Now we are returning the additional serverException with the error information, that provides the following entries:

    • exception.type: the full exception name of the topmost exception.
    • exception.message: the message of the topmost exception.
    • exception.where: the stack trace, only in debug mode.
    • rootException.type: the full exception name of the root exception.
    • rootException.message: the message of the root exception.
    • rootException.where: the stack trace, only in debug mode.
    • exceptions: an array of elements containing the type , message and where with the stack trace for all exceptions in the exception chain.
  • As always, more tests have been added to help make DJN code more robust.

To download version 2.2, go to the DirectJNgine googlecode site.

Javascript remote logging with Log4j & log4js-ext

I released log4js-ext several days ago, and I’m planning to add remote logging support. I would love to consolidate client and server side logging at the server to help me debug complex browser/server app interactions.

The first logging system I would like to support is Log4j, with slf4j a close second.

Log4js-ext remote logger sends information that I will gather in my RemoteLoggingEvent Java class, as follows:

public class RemoteLoggingEvent {
   String level;
   String category;
   String formattedMessage;
   String ndc;
   long   timeMillis;
   boolean hasLoggedObject;
   String formattedLoggedObject;
   String formattedTime;
   // ************************************************************************
   // javascript side information ommitted on purpose because there is not
   // much the server can do with this raw data
   // ************************************************************************
   // Date time;
   // String message;
   // Object[] formatParams;
   // Object loggedObject;
   // int levelLevel;

   // ...

My first thought was to simply redirect the data to a Log4j logger, much like this:

Logger logger = Logger.getLogger(remoteLog.getCategory());
logger.log( getPriority(remoteLog.getLevel()), remoteLog.getMessage() );

This works, but there is a *BIG* but. The logged timestamp will correspond to the moment when we called the log method at the server, not the client side timestamp.

Due to the asynchronous nature of web requests, it might well happen that two logs A and B that were created in that order at the client could be received in the server in B, A order. Then the call to logger.log will generate slightly different timestamps, telling lies about the ordering of client-side actions. Really bad for debugging purposes.

It is important to note that I’m implementing remote logging with an eye to help debug during development. During development we tend to run the client an the remote app in the same computer, and then we have the guarantee that the client and server side timestamps are coherent. In that scenario, using the client generated log timestamps will be a real boon, providing correct sequencing information.

That can be so useful for debugging that I decided to byspass the Log4j loggers and log directly against their appenders, which have a nice doAppend method that receives a LoggingEvent accepting not only messages, but a timestamp, an NDC, etc.

The following code will create a LoggingEvent based on the remote information provided to the server in my own RemoteLoggingEvent object:

private static LoggingEvent adaptLoggingEvent(RemoteLoggingEvent log, 
      Logger logger) 
   long millis = log.getTimeMillis();
   // A configuration setting very useful during development
      millis = (new Date()).getTime();
   String message = getMessage(log);
   ThrowableInformation exceptionInfo = null;
   if( log.getException() != null ) {
      exceptionInfo = new ThrowableInformation(log.getException());
   LoggingEvent le = new LoggingEvent(
         log.getCategory(), logger, millis, getLevel(log.getLevel()),
         message, log.getThreadName(!APPEND_NDC_AT_THREAD_NAME_END), 
         log.getNdc(), null, null );

   return le;

Now, the problem is Log4j loggers do log following certain rules, and calling its appenders doAppend method would bypass them all.

To follow the rules we have to take into account several things:

  • We have to log to appenders *only* when the log level/priority (error, warn, info, etc.) is greater than the logger priority.

  • We need to write to the logger appenders, and then to all parent appenders, and to the parent parent’s appenders, and so on. But *only* if the current logger additivity is set to true, else the only appenders I have to write to are those directly attached to the logger (maybe none).

Here is the code that takes all of this into account:

public void log(RemoteLoggingEvent log) {
   Logger logger = getLocalLogger(log);
   LoggingEvent le = adaptLoggingEvent(log, logger);
   Category cat = logger;
   // Because we are not logging through the Logger interface, we 
   // need to take care of logging only when level is ok
   if( le.getLevel().isGreaterOrEqual(cat.getEffectiveLevel())) {
      // We traverse all appenders upwards -unless additivity is set to false
      // This simulates calling Logger.log, which we don't call because 
      // it would create the wrong timestamp, the wrong 'thread', etc. 😦
      while( cat != null ) {
         @SuppressWarnings({"unchecked" })
         Enumeration<Appender> appenders = cat.getAllAppenders();
         while( appenders.hasMoreElements()) {
            Appender appender = appenders.nextElement();
         if( cat.getAdditivity()) {
            cat = cat.getParent();
          else {
             cat = null;

I wanted to get the same level of support for slf4j, but I just found that there is not an easy way to log timestamps and other additional information. In the end I just wrote the slf4j adapter by calling, error, etc., with the corresponding timestamps being server generated.

This will make into log4js-ext in the following weeks, probably using my own DirectJNgine to perform remote communication, so that I can benefit from the built-in support for batching, etc.

log4js-ext published

As of lately I’ve been having some extra fun with ExtJs, and I decided I needed a serious javascript logging library for ExtJs. Something that closely resembled log4j or the thousand java logging libraries out there.

When I made a list of desired feature the logging library should provide, this is what I got:

  • It was a must to have a good GUI based viewer that allowed me to filter logs by different criteria: category, priority (info, error, debug, warn…), message content, etc.

  • I needed some way to group related logs so that I could distinguish what logs were part of a certain complex operation. I decided to support the NDC concept Log4j supports. Of course, the GUI viewer allows NDC filtering.
  • I needed to log arbitrary objects and see them as nicely highlighted JSON. And, of course, I should be able to search any value in the logged objects with a viewer.

  • I needed an easy way to format log messages, avoiding hand-made concatenations and formating as much as possible.

  • I wanted it to be extensible, meaning I should be able to add support for things such as remote logging, to define new ways to format messages, etc.

  • I wanted it to be easy to use and familiar to users of popular log libraries for other languages, such as log4j, logback, etc. Log4js-ext closely resembles log4j.

  • And, of course, I needed robustness. This means, among other things, lots of automated tests.

In the end, I decided the best thing I could do was to implement my own utility. It is a small project, worth 3 or 4 days of work to have it fully tested and production ready, and therefore feasible.

Now that the project is finished, the standard log viewer looks like this -well, much better, due to image scaling!

Standard log viewer

If you don’t like this, you can get things logged to your browser console (or both). This is what it looks like for Chrome:

Log output in Chrome's console

I hope you notice the object logging capability built-in, just look at the last logging entry in both viewers. What’s nice is that you can search for values in the logged object, a must when you have tons of logging output.

I would like to thank the author of the JSON highlighter I used to make the logged object output more readable, though I don’t know who he is. I just found the code here, so I’ll assume the guy that gave the answer is the author. Again, thanks a lot!

The version 0.9 of the library is available in GoogleCode.

The library is quite robust, and it has more than 25 automated JsTestDriver tests (it is small, to be true), but I decided to publish a 0.9 version to get some feedback before committing to a 1.0 version.

I expect to have 1.0 out as soon as possible, as the only things missing are some healthy extra user feedback and some code refactoring I’m undertaking right now.

Tests with JsTestDriver

BTW, here are the JsTestDriver test reports, I just can’t help it…

The test results for version 0.9

I have to say that I find JsTestDriver very interesting. I have still not selected my definitive TDD tool for Javascript, but JsTestDriver is a strong contender right now.