Script injection in web applications, which also goes by the name XSS (Cross Site Scripting), is a serious and common security problem. It arises when the application does not appropriately escape data that comes from an untrusted source. The data could contain malicious JavaScript that would get executed. If developers are burdened with remembering to escape each and every instance of data they will eventually forget or make a mistake and a security hole will result. One main point of the article is that string interpolation should be automatically secure — it should do the right escaping for you.

Every now and again the issue of how to escape special characters when generating HTML files comes up on the StringTemplate mailing list. The answer is very straight forward – write a renderer. My main point, which I’ll get to eventually, is about the details of the renderer.

The author of the above article distinguishes string interpolation from “full blown” templating languages. JSP, PHP, and XSL were given as examples. Three problems with full blown templating languages were given (folks familiar with StringTemplate know there are more):

1. They do nothing to help solve the escaping problem.
2. They are verbose and add lots of boiler plate.
3. They don’t make simple things simple.

I think StringTemplate holds up well against these complaints. The renders can do escaping. The simple templates (.st files) have no boiler plate at all and the group files (.stg) add only a little. Lastly, it doesn’t get much simpler than “Hello $name$!”.

The most important issue and the one the article is really about is number one — escaping.

On an HTML page what characters need to be escaped depends on the context: element content, attribute value, JavaScript, and URIs. StringTemplate rendering assumes a single rendering context. As of StringTemplate version 3.1 the format option can be used to specify a specific rendering (format). The trouble with this is that the template writer must be aware of the different contexts and what format to use in each. An alternative is to come up with a one size fits all escaping renderer. This will end up escaping more than it needs to. There may be some fringe cases where this could cause problems.

The thing I found most interesting about the approach given in the article is that the context scanner keeps track of the current context in the output so that the escaper can do the right escaping according to the context. To put this in StringTemplate terms the output after rendering would be feed into a parser that tracks changes in context (element to attribute value to JavaScript to element content etc.) and informs the renderer(s) of the current context.

You can still use the format option or other custom renderers to specify how the data is turned into a string (for example a number could be currency or a percentage). This is something that the template author is expected to know. It seems reasonable to me that it would be better if the author didn’t need to understand the intricate escaping rules of web pages.

A key innovation of StringTemplate is the identification of the Model-View-Controller-Renderer pattern — that there is a renderer distinct from the view. (Read about MVCR here.) Now I am seeing the render as two distinct pieces. There is the format renderer and the escaping renderer. The former is concerned with if negative 23 should be output as (23) or -23. The latter is concerned with if single quote should be output as ‘ or ' The format renderer runs first and passes its output to the escaping renderer.

The reason it makes sense to think of two different renderers is that they each have different masters. The format renderer is directed by the needs of the application, the locale and perhaps presentation directed formatting with the format option. The escaping renderer’s function is determined by the language of the output: SQL, HTML, XML or just the text of an email message.

Other thoughts and observations:

Is there a need for a JavaScript implementation of StringTemplate? As it becomes more common to move the model, view and controller to the client there could be a role for it. It could also run server side in Rhino. Although the author of the article states that it is not a goal to create a template language I wonder if the extra capabilities of StringTemplate would be welcome.

The method used in the article for implementing auditable exemptions (cases where you know the data is safe and you don’t want it escaped) is the same as would be done in StringTemplate. Just wrap the data in a type that returns the raw string when rendered.

In the article a simple FSM was used to parse the HTML. Only an excerpt of the code was shown and I didn’t bother to download it and look deeper. It seems to me that the output HTML needs to be well formed otherwise the parser would be very complicated if it has to deal with the tag soup that browsers deal with.

This is now at least the fourth time I’ve seen a parser written in JavaScript so I think there could be a need for a JavaScript backend for ANTLR. I would have liked to create a l-system language parser in JavaScript for a recent project of mine. Is there already an ANTLR for JavaScript that I don’t know about?

Does anyone else think that it would be useful/feasible to have a HTML escaping renderer (that does the correct escaping in automatically) as part of the StringTemplate library?

About a month later I went to the ajax experience conference and learned about the new HTML canvas element among other things. I wanted a project to get practice writing JavaScript, more experience with using jQuery, and to try out jQueryUI and canvas. L-systems seemed like a great project for this purpose.

Before I get to my project here is a real quick definition of what an L-System is. A Lindenmayer system (L-system for short), named after biologist Aristid Lindenmeyer, is a rewriting system. It takes a list of symbols and repeatedly generates a new list of symbols by following a set of rules. Each new list of symbols is called a generation. The symbols can be given a graphical representation to produce an image algorithmically. The beauty of L-systems is that with just a few symbols and rules amazingly complex images can be created. L-systems were developed to model plants but can also draw fractals. More information is available from the links above and from the lsystem.js documentation.

My project, L-system.html, is a serverless web application for exploring a number of plant and fractal models created with L-systems. You select a model, change its parameters such as angle or distance depending on the model, set the number of generations then press the play button to see it drawn.

Being able to adjust the parameters of a model helps to understand it. When you see a fractal of some reasonably high number of generations it is not always clear what process gives rise to it. Stepping through from the zeroth to first and second generation of a fractal really helps to visualize the rule that makes the fractal what it is.

L-system.html is not very fancy or even complete. The main purpose for me was learning canvas, jQuery, and jQueryUI. It makes use of two reusable JavaScript libraries that I wrote:

• lsystem.js – a L-system library that supports parametric or non-parametric, deterministic or non-deterministic, context sensitive or context free, and branching or flat models.
• turtle.js – A 2d turtle graphics library on top of the HTML canvas API.

Both of these are open source with a BSD license if you care to use them. They can be used together or independently. You can download the whole project. If you make improvements to them or have suggestions I’d love to hear about it.

The rest of this post covers what I discovered along the way.

First I have to say I’m impressed with canvas and its capabilities. Its not as full featured as most mature graphics libraries and there are a few features that are not supported on all browsers but it opens up another area to web applications (without having to resort to Flash) that was previously only found in desktop applications. People are doing cool things with it already. Support for IE is accomplished with explorercanvas from Google. I have tested my project on Firefox 2, IE 7, Opera 9.24 and Safari 3beta (on Windows) and it works well except that the save functionality doesn’t work in Safari or IE. Also I have noticed IE to be slower than the others but have not done any rigorous timing.

Thanks to Canvas Paint (a canvas implementation of Microsoft Paint) for the technique for saving canvas images.

L-system.html is an example of a serverless web application. Its an application because it is interactive and does something at least a little interesting or useful. Its a web application because it runs in a web browser. Serverless is a somewhat new category of web application. It basically means that none of the application behavior is running on a server. A web server may be serving the HTML and related resources but there is no servlet, or PHP script, etc. A server may be providing raw data or persisting data (like the Amazon S3 service) but this is just data storage or acquisition and often the servers or services are not even owned by the creator of the application. In some cases the application can just be an HTML file on your local file system. L-system.html can run this way. An even cooler example of the kind of interesting things that can be done without a server is TiddlyWiki – a serverless wiki.

After working more with jQuery I’m convinced that this is the way I want to write JavaScript and to work with the DOM. However I’m not anywhere near as impressed with jQueryUI. It just doesn’t seem ready for prime time. I only wanted a few UI elements: a slider, a dialog, a progress control and a color picker. There is no progress control or color picker in jQueryUI although there are plugins for color pickers and maybe a progress control as well. I thought it would be best to stick to the official UI widgets. I struggled to learn the UI widgets and get them to work the way I wanted. I needed to read the source as well as the documentation to figure them out.

The first problem I ran into was with the slider on on IE 6 and 7. I found a bug report that said the problem was fixed in the latest. After downloading the latest from the SVN source repository the problem was indeed fixed.

I wanted to be able to move the slider with the right and left arrow keys and to show that the slider had focus. I know that jQueryUI doesn’t have full accessibility support yet, but keyboard support seemed like it should work for a control like slider. I noticed some code to turn on keyboard support when webforms (not sure what that is) is present. This helped me figure out how to get it working the way I wanted. You can tab to or click on the slider to give it focus and then use the arrow keys to move the slider.

One thing that bugs me about some ajax apps is that they break the back button. I really wanted to get back button history working with jQuery in this project. At the ajax experience conference I saw the history plugin demonstrated and it looked so easy. So far my attempts to get it working have failed. The history plugin seems like it would be easy to use with the tab control but i’m not using the tab control. Then I read that the new tab control in jQueryUI doesn’t work with the history plugin. There is a mention that the history plugin needs to be rewritten but no time frame for when it will be available. Since I’m not using tabs it may be possible to get the current history plugin working but I haven’t figured it out yet.

My next problem was with the dialog control. I found that I could end up with multiple copies of the dialog displayed. I decided to try the latest version from SVN like I did with the slider. I did manage to solve my problem by using the new dialogInit method. (There may have been another way.) The trouble is that at the time I got the latest version of dialog the mouse management was rewritten which broke the slider and resizing of dialogs. As of this writing they still are not fixed. There may be some set of source versions that will solve both my slider and dialog problems but I don’t have time to find it. So at this point the sliders do not move with the mouse (you get a JavaScript error). I’ll just have to wait for the jQueryUI mouse implementation to get finished.

I noticed that some of the jQueryUI documentation (at least the examples) are in advance of the current 1.0 release. For example, the dialogInit method was shown in one of the dialog examples even thought it is not in the 1.0 release.

There is no progress control in the jQueryUI so I made my own. The next step will be to learn how to turn it into a plugin.

Along the way I found a number of useful tools and practices for writing JavaScript

• Firebug is a very good tool. I knew that already but with this project I used the profiler for the first time. The profiler is very easy to use and helped me to improve the performance of the L-system algorithm. I also used the console log, which Firebug lite makes work cross browser.
• I’m using JSLint from an ant task and it has found some problems. I described the technique for ant integration here.
• I used jsdoc-toolkit to generate the library documentation. This tool is very easy to use and produces nice results. Since the JavaScript is going to get minified anyway I think it is a good idea to keep the documentation in the source.
• For testing the 2 library modules I used jsunit. Actually I only used the assertion functions from the core. This looks like it could be a good tool but I need to learn more about how to use it effectively.
• I use just the CSS library from YUI.
• To build the final html as well as the aggregated and minified resources I use ant, yui compressor, and STST. One thing I noticed is that yui compressor catches some JavaScript problems that JSLint did not.

After I was just about done with this project I discovered that someone else created a canvas L-system implementation LSys/JS. Although it doesn’t support as many kinds of L-systems it does allow you to enter your own. Also it does not have a reusable library layer.

If you have read this far without trying it out you should take a look at L-system.html now. If you want to add your own L-system models you can download the project and modify the lsystem.html file. If you come up with a cool model leave a comment with a link to it.

Programming-in-the-large is about how to deal with creating and maintaining large programs. Of particular interest are the facilities a language provides for modularization and how the modules interact. These interactions may touch on issues of information hiding and the abstractions available in the language.

StringTemplate allows you to organize your templates in different files. There are two types of template files: the simple template file and the group file. The simple template file is often just called a template although this can be ambiguous because groups contain templates and templates can even be created programmatically. I use the term simple template file after the XSLT term simplified stylesheet module which is a stylesheet that starts with a literal result element.

Simple templates are just files with StringTemplate actions between delimiters ($ $ or < >). There is one template per file. They cannot declare arguments, the name of the template is the name of the file (less the extension) and the extension must be .st. Simple templates can invoke (also known as call, reference or include) other simple templates by using a relative path to the other file. For example:

/dir1/template1.st:
  This is the caller. $dir2/template2.st()$
/dir2/template2.st:
  This is the callee.

Group files allow you to define several templates in a single file. The group file must have a .stg extension. Group files also support inheritance, and interfaces (more on interfaces later). The templates in a group file can declare arguments. Templates can invoke other templates in the group file or any of its super groups. For example:

supergroup.stg:
  group supergroup;
  util(arg1) ::= <<
  This is in the super group. $arg1$.
  >>
subgroup.stg:
  group subgroup : supergroup;
  template1(arg1) ::= <<
  This is in the sub group.
  $util(arg1)$.
  >>

Now here is where implementation details get in the way. For simple templates to invoke other simple templates they must be gathered together in a StringTemplateGroup object. This object is given a root directory and all template files must be under that root. This same object when constructed differently represents a group file. Templates in one group cannot call templates in another group unless the groups are in the same inheritance hierarchy. This means that templates in a group cannot call simple templates and vice versa. (Technically it should be possible to programmatically cause simple template groups and group file groups to inherit from one another but it can’t be done from within the group or simple template file syntax.)

I have been using StringTemplate to generate the HTML for a web application. I started out using simple templates but soon found I wanted the ability to declare arguments on my templates so I switched to the group file format.

The program calls a template that will produce a complete web page. That template relies on other templates to generate parts of the page. These other templates in effect represent controls — aggregations of HTML such as an input field and its label, an image with a drop shadow, or a table with next and previous links. I use one group file for each page. The controls are reusable so they shouldn’t go in the group for a specific page. The only solution available is to put them all in a base group and have the page groups inherit from that base group.

There are a number of problems with this. The first is that all the control templates must be put together in a single group (because a group can only inherit from one base group). This group file quickly gets large. Second it forces an is-a relationship where there isn’t one. If my base group is called “dumping-ground-for-all-controls.stg” and my page group is called “accounts-page.stg” it doesn’t mean that an accounts-page is a kind of dumping ground for all controls as inheritance would imply.

I should be able to choose my own way of organizing my templates into groups. I would much rather put each control in its own group or possibly group like controls together. My page group would then call templates in any of these group files.

I wouldn’t use inheritance unless one page or control was truly a specialization of another page or control.

To get a better understanding of the StringTemplate notions of templates and groups it is useful to compare them with the abstractions of other languages.

Conceptually a template is like a function. It takes a collection of named attributes and returns a text string. Simple template files and group files are modules. They are the building blocks of a StringTemplate text generation system.

Languages like C allow you to group your functions together into files (modules) any way you like and functions can call functions in other modules. StringTemplate should allow the same.

On the surface group files may seem like classes. They have some things in common but also important differences. A group is a collection of related templates in the same way a class is a collection of related methods. The thing that relates a class’s methods is access to and operations on a common set of data. This data is an object or class instance. In StringTemplate there is no mutable state so it makes no sense to speak of class instances or objects. Groups support inheritance like classes do but template polymorphism is a little different from traditional OO languages. In OO languages like Java or C++ the method called depends on the type of the object the call is made on rather than the type of the object reference. In StringTemplate, again, there are no objects so the template called depends on the group used to make the initial template call. It is a good thing that the StringTemplate group abstraction was not called a class lest someone take the analogy too far.

StringTemplate groups do a good job at providing needed capabilities for programming in the large. Specifically grouping like templates together in a single file for convenience, declaring template arguments, inheritance and template polymorphism, and interfaces. They fall short in assuming that all templates needed by a text generation system belong in a single group hierarchy.

It may seem restrictive for simple templates to limit the file to containing a single template. Imagine C forcing you to have one function per file. One might be tempted to get rid of the simple template and just keep the group file. However there is value in having simple templates in addition to groups. The simple template is useful when the output is mostly static text with just a few holes that need to be filled in with data. The simple template is like a form letter (Dear $name$, You may have already won a million dollars…). These templates are the easiest for non-programmers to understand because there is no extra syntax beyond the template actions and the template looks very close to what the final result will be. The same argument for the existence of simplified stylesheet modules in XSLT can be made for simple templates.

I said I would come back to group interfaces and this seems like a good time to do it. I think StringTemplate gets interfaces exactly right. An interface provides a contract between the caller of templates (the program) and the templates themselves. The interface defines what templates the program plans to call and that the group must implement.

Interfaces in StringTemplate work very similar to Java interfaces. The interface specifies a set of template names and their arguments. When a group implements one or more interfaces it is agreeing to implement all the templates from the interfaces with the right number of arguments for each. An error is given if the group doesn’t implement the templates specified in the interfaces. If the writer of the program changes the interface then the writers of the group files will know that they need to make corresponding changes.

Another difference between groups and simple templates is in their ability to reference other groups or templates in different folders. Simple templates can use relative paths to reference templates in other folders. Groups can not use relative paths to reference super groups, or interfaces. However the implementation allows specifying multiple directories to search for super groups and interfaces but simple templates must be under a single root. I view these as implementation details and not limitations that are inherent in the language.

To better support programming in the large I would like to see StringTemplate meet these requirements:

• allow multiple groups during template processing
• allow mixing simple templates and groups. A simple template can call a template in a group and a template in a group can call a simple template.
• support some kind of namespaces for simple template, group and template names

Here is a proposal for meeting the above requirements. Groups and simple template files, collectively modules, are kept in a multi-rooted hierarchical store such as a file system. Other stores such as a database or compressed archive could be used as well — even a mix of them. The StringTemplate engine implementation would provide a way to specify the roots where modules are found. Something along the lines of the Java class path would be one possibility. The implementation could borrow some ideas from the Java ClassLoader. I’ll call the hierarchical containers of modules folders. They could just as well be called packages or directories.

Example:

controls/listview.stg:
    listView(id, caption, data) ::= ...
    listCell(item) ::= ...
    ...
controls/form.stg:
    button(id, value, label, accessKey) ::= ...
    checkbox(id, value, label, accessKey, checked) ::= ...
    ...
pages/users/list.st
pages/users/add.st

The above example defines two groups (listview and form) and six templates (listView, listCell, button, checkbox, list, and add) in two folders (controls and pages).

A group file would introduce a kind of virtual sub folder that contains templates just as a normal folder contains simple templates. The fully qualified name of a template is the forward slash separated list of folders, possibly including a trailing group name and finally the template name. For example the template pages/users/add would add a button to the page by referencing $controls/form/button(...)$ and template pages/users/list would reference the listView control as $controls/listview/listView(...)$. The listView template would reference the listCell template simply as $listCell(...)$. The general rule is that templates in the same folder or group don’t need any folder or group path in front of it. If part of a listView control included buttons then the listView template would refer to the button template as $controls/form/button(...)$.

The hierarchy of folders provides a kind of namespace similar to Java packages. Folder, simple template, and group names are in the same namespace. You can’t have a folder, group, or template with the same name in the same folder. Even though the file system will allow a folder to contain sub folder foo, and files foo.st and foo.stg this would not be allowed. Folders can contain sub folders, groups and templates. I seen no need for groups to contain other groups (as in Java’s inner classes).

I used the ‘/’ character rather than ‘.’ to separate folders, groups and templates because ‘.’ is already used to specify nested attribute data and because it is already used in simple templates.

The syntax of group files and even simple templates should be extended to support importing folders or groups so that the templates in those groups and folders can be used without having to use fully qualified names. Different languages have different functionality for one module specifying other modules it intends to use. In StringTemplate I think it should be possible to specify a specific template in a group or folder or all of them. It should be possible to provide an alias for a template.

As for the syntax I’m not sure what to do. It might look something like this:

• $use controls/listview/*$ to import all the templates from group listview.
• $use controls/listview/listView$ to import just the listView template from group listview.
• $use controls/listview/listView as list$ to import the listView template but call it as $list(…)$

In a group file the use directive could go at module scope without being enclosed in the action delimiters. At module scope it would apply to all templates in the module. For example:

  group foo;
  use bar/template1;
  ...

Another important principal in OO languages (and even languages like ADA) is information hiding. Classes can declare their members, data or methods, as public, private or protected. Since template groups have no mutable state there is no data to be hidden.

I think it may be useful to specify templates as being private. Some templates are just a useful way of implementing another template and don’t need to be called from outside the group. The implementation of one template can be broken down in to a number of sub templates. The details of which are not important to the callers of the main template. The main template is public while the sub templates should be private.

One possibility would be to add keywords like public or private to template definitions in a group file but there is no elegant way to do the same for simple templates. I think it would be fine to use a naming convention such as template names beginning with an underscore can only be used from templates in the same folder or group.

I believe these enhancement will make reusable template libraries possible. Even if there isn’t much need for template libraries a large project will benefit from being able to organize the templates in a way that makes sense for the problem at hand. You can choose to use simple templates or groups depending on what is best for each template rather than based on rules for what calls are allowed.

StringTemplate the language

Presently StringTemplate is described in terms of both its syntax and implementation (API). I think it is important to describe the StringTemplate language independently from its implementation. I see the following benefits to doing this:

• It allows thinking more clearly about the language without implementation details affecting language design decisions.
• It promotes independent interoperable implementations either for additional host languages or alternate implementations for currently supported languages. Implementations could be interpreters or compilers.

A consequence of describing the StringTemplate language is that the language should be versioned independent of the implementation.

Implementation details include locating, loading/parsing, and invoking templates, providing data to the templates, output filters and rendering.

For the most part the language specification can state that an implementation will provide a way to do X without having to specify how. For example, it is up to the implementation to provide a method for invoking a template.

Some things will be dependent on the host language for example the character set encodings supported for templates. Even though there is no syntax in SringTemplate for specifying the character set encoding the language specification should state any underlying assumptions or dependencies.

Data model

One very important thing to describe independent of the implementation is the data model that StringTemplate operates on. StringTemplate should clearly define its data model and then, separately, the implementation would define how it maps native types onto that data model. Although a template knows nothing about the type of an attribute (with a minor exception for booleans in if expressions) it is very aware of the “shape” of the data. StringTemplate supports data with these shapes: scalars (single value), lists (ordered collection), and maps (unordered keyed values). It also has a special value called null that produces no output, and evaluates as false in if expressions.

Lists can contain any mix of scalars, lists, maps and nulls. A map entry value can be a list, scalar, map or null. The keys of a map must be scalar strings.

The ability to represent the data model literally in StringTemplate is currently inconsistent.

There is a literal syntax for strings, which makes sense since everything gets turned into a string in the end. The grammar includes a definition for integers (INT in action.g) and this should be removed. There is no reason why StringTemplate should know anything about numbers of any kind. I think Terence agreed to this.

List literals can only be defined in the context where an expression is expected and are unnamed but maps can only be defined in group files at the file level and they are named. I don’t see the harm in allowing literal maps, lists and strings at both the group file level and in the context of an expression. One problem is that literal maps use square brackets as delimiters just like lists. I’m not sure what the best syntax for literal maps should be. Curly brackets are already used for anonymous templates, square brackets are used for lists, parentheses are used for template argument lists. Perhaps [: could introduce a literal map. At the group level you should be able to name a list, map or string like so:

  group example;
  map ::= [: "key1": "value1", default: "none"]
  list ::= [ "1", "2", "3" ]
  string ::= “something”
  …

The capabilities for literal data should be similar to the JSON
format even though the syntax will likely need to be different. The exception is that numbers and boolean types are not supported.

The program can define a list of lists (for example to represent a matrix) but the literal syntax doesn't preserve the nesting. Even though $table(t=[ [ "a", "b" ], [ "c", "d" ] ])$ is allowed it gets flattened. Not sure if this is intentional or a bug but I think that the literal structure should be preserved. If there is a need to flatten a list then it should be made into a function such as $flatten(list) : ...$. Once there is a literal syntax for maps it should be possible to compose a literal map of lists and a list of maps.

There should be a literal syntax for null so that you can create lists (or maps) with nulls in them and explicitly pass null to a template.

Map and object unification

Maps and objects should be unified. There are a few differences between how StringTemplate deals with objects and maps. Ideally from the StringTemplate data model point of view there should just be one thing; call it a map or call it an object it should behave the same. When you write $A.B$ you have no idea if A is an object and B is a property or A is map and B is a key. In fact the program should be able to change the implementation from an object to a map or the other way around without the template being affected.

The differences are: 1) Maps support the pseudo keys "keys" and "values" and objects do not. 2) Objects have an implied call to toString that maps don't. 3) They have different behavior when a property/key doesn't exist. These differences make the template aware of the underlying implementation type.

I never did like these pseudo keys because they pollute the namespace of map keys. Also there is no reason why you couldn't ask for all the properties of an object. The "values" key is not really needed because $map$ has the same effect as $map.values$. To solve these problems I think values and keys should be functions like first and last. $keys(A)$ will return a list of all the keys of a map or the properties of an object. For a scalar it should probably return an empty list and for a null return null. The values function would be defined similarly except that it would return a list of the values of all the keys or all the properties.

The remaining two inconsistencies (access as a scalar and access a key that isn't present) are trickier. I'm not too sure what the best behaviors are.

There is good reason to support accessing an object as a scalar and providing access to its properties as well. Imagine a date object. When you access it as a scalar it should result in the text representation of its date value. But it may also have useful methods such as month and day. Example: $today$, $today.month$. For a map I'm not sure if $map$ should return null (empty string) or a list of all its values as it does now. Either way a fundamental differences remains between maps and objects. Hopefully it isn't too surprising; this extra ability of objects.

It would be nice if you had control over the behavior when a property/key doesn't exist in an object/map. I'm not sure if the program or template should get to control it and if it is the template how it would be specified. The literal map syntax already has a default keyword to specify the value when a key doesn't exist. I think the default should be to give an error if an object property or map key doesn't exist. For either a map or an object there should be a way to specify the default value for a non-existent key/property. The value of the default (as well as the values in the literal map syntax) should support template expressions. The implicit attribute key would be available to the template. This is more powerful than the key keyword currently supported in the literal map syntax.

One way to support adding a default specification for maps/objects would be composing maps from other maps. Suppose [: map1, map2... default: expr ] was a literal syntax for composing a new map from map1 and map2. The action $foo(arg=[: myMap, default: "none" ])$ would create a new map and add all the key value pairs from myMap to it and also specify that if a key is missing the string "none" is returned. This map is then passed as the value for argument arg to template foo. There may be a better way to specify the non-existent key behavior.

Another possibility might be an option like the null option. It might look like this: $map.optionalkey;notexists="none"$

Template argument shape

Clearly there is no point in specifying the type of arguments in template definitions when templates have no knowledge of the underlying attribute type. Example: myTemplate(int i, Date d) ::= ... makes no sense. But would it be useful for a template to declare the shape of the argument value (scalar, map, or list)?

This may be similar to what Terence wrote in this post:

I have been thinking for quite a while about adding *, +, and ? to the definition of template arguments so that you know whether you should get zero more, one or more, or zero or one values in the attribute. That way ST can check the cardinality and existence
automatically for you.

It seems that this would only apply to lists and would tell something about how many items are expected in the list.

If a template definition declared that its argument must be a scalar and a list or map is passed instead an error would be given. The distinction between a scalar and an object/map is very subtle. The declaration is more about how you intend to use it rather than the actual underlying type. In one case you expect it to have some specific properties and in the other case you don't care if it has properties or not.

The shape declaration could be optional. I'm not sure if this is useful or not. At a minimum it would help document what the template expects. Which leads to the next topic.

Agreeing on the data

Every now and then the question of "how can I find out what attributes a template uses" gets asked on the stringtemplate-interest mailing list. From the other vantage point a reasonable question is "what data does the program make available to the template". The way you look at it may come down to who is driving the application design; the template writer or the program writer. If they are the same person it may depend on how they see the problem; either driven by the needs of the output or driven by the program and the data it has.

Either way the important point is that the program and the template have to agree on the data and need some way to communicate what the data looks like. Currently StringTemplate has no way to facilitate this communication or validate that the data meets the agreed upon structure. I'm not saying that it needs to. I'm just wondering if it would be useful and what it might look like.

In my own work so far I have not done anything to write down what the structure of the data is. It has not been a problem for me because the application is small and I am writing both the program and the templates but I know this won't scale. Simply looking at the template arguments is of little help because the data can be deeply nested. Even the above mentioned suggestion to declare the shape of the template arguments doesn't help much because of this deep structure. How do you solve this problem? (Thats not a rhetorical question - I'd like to know what you do.)

In the XML world a schema is used for this purpose. The producer and consumer of some structured data both agree on the schema. The schema describes the structure of the data as well as the types of individual values. Any time between when the data is produced and when it is consumed the data can be validated against the schema. With XML the data is usually serialized in a text document in XML format but it doesn't need to be. The schema language may be Relax NG (unless you are unfortunate and have to use XML Schema).

This makes sense for XML but may be overkill for StringTemplate. A big difference between the two is that the coupling between the producer (the program) and the consumer (the template) is much tighter with StringTemplate. The template and the program are designed and built together and the data is unlikely to be serialized in between. Still I wonder if something like Relax NG could be useful as a way to communicate the data agreement. Validation of the data could be done as part of a unit test. Another potential benefit of a schema would be assisted editing. The editor would know about what attribute and attribute properties are available.

In the Java/OO world the class declarations describe the data. The data structure may be given as a UML diagram. This works for communication. Within the program code Java's type checking can help ensure that the data is valid but there is nothing to enforce, at "build" time, that the template is using the data according to the Java types. Again, I'm not suggesting that StringTemplate needs strong or static type checking - just making a comparison.

Possible host languages

Thinking about StringTemplate as an independent language has me wondering what kinds of host languages could it be implemented in. Are there any languages or classes of languages where it wouldn't work well, isn't needed, or would have limitations.

One issue that comes to mind is that the rendering of attributes depends on the type of the attribute. The identification of rendering as a distinct aspect of template processing is a very powerful feature of StringTemplate. The template doesn't need to know about data types or type specific formating because the renderer knows how to format the attribute according to its type.

What would happen in a dynamically typed language, would the renderer know how to format the attribute in a meaningful way? If not then the template would need to rely on the format option.

Well that's more than enough thoughts for now. I hope to get the next batch written up soon.

StringTemplate is both a language for describing text generation and a tool (template engine) that implements that language and provides an API to connect the tool to another program.

StringTemplate is not a general purpose language and cannot stand on its own. By design it has no way to perform general computation. It also has no general way to create data and limited ability to specify literal data. It is up to some driving program to produce and or make data available to the StringTemplate template engine. This is not a negative – this is it’s strength. See this excellent paper for why.

However, the lack of readily available data makes learning the template language difficult without also learning the StringTemplate API. In order to try StringTemplate out one must learn the template syntax and semantics, and the API and then write a program to produce some data and call the template engine.

In theory one benefit of the separation of presentation from business logic is that a project can be divided between people who work on the business logic (the program) and people who work on the presentation (the templates). The people working on presentation don’t need to be programmers.

The current documentation for StringTemplate describes both the template syntax and the Java API at the same time. In some cases it is confusing to learn two things at once and more importantly it excludes non-programmers from learning StringTemplate.

In contrast, trying out XSLT is is easy (even if mastering XSLT is hard) because all you need is a text editor to create both your input documents and your style sheet and an XSLT program to process the style sheet with the input documents. Likewise trying out HTML is easy, all you need is a text editor and a browser. Interpreted languages like Python that have an interactive mode also make it easy to jump right in and get the feel of things. (I’m not saying that StringTemplate is like XSLT, HTML or Python, I’m just comparing the ease with which one can dive in and try the technology out.)

So what StringTemplate needs is a literal representation for data that can be used as input and a stand alone program to read the data and templates to produce the output text.

I choose JSON as the literal data format. The reasons are:

• Its a standard
• It is becoming increasingly well known
• It is easy to type in and not overly verbose
• I had already made changes to get a Java JSON library to play nice with StringTemplate.
• It supports all the data shapes that StringTemplate works with.

I created a command line tool called StringTemplate Standalone Tool (STST). It takes as input the name of a file containing data in JSON format, the name of a template and optionally the name of a group as well as various options.

Look ma, no programming.

Here is a simple example of how it is used.

Create a JSON file called data.js with this content:
{ "audience": "World" }

Create a StringTemplate file called first.st with this content:
Hello $audience$!

From a command prompt in the same folder as the above two files type
> stst first data.js
The output is
Hello World!

It also works with group files and can use the angle bracket lexer. For command usage type: stst -h

The next thing StringTemplate needs is a tutorial that focuses on the template syntax independent of the API. STST enables this kind of tutorial. Separate documentation can focus primarily on the API.

Download STST here

This tool can have other uses as well. Here are some ideas:

• It is a quick way to try out template syntax without having to write or modify a program. This can benefit seasoned template authors as well as beginners.
• It could be part of a test framework for StringTemplate itself or for a set of templates in an application. The templates can easily be processed with known data and the expected results asserted.
• It enables development of the templates before or in parallel with code by creating a JSON data file that represents what the code will produce.
• It could be used as part of shell scripts where some existing program outputs JSON data and STST is used to generate text from it. The data could come from a web service for example.

What could you use it for?

Future possible directions for STST

• Integrate the functionality into an IDE for StringTemplate
• Expose the functionality in an interactive servlet (web app).
• Support other data formats like CSV, YAML, or XML
• More support for renderers

The article Localization and Internationalization with StringTemplate shows an example using the java.util.Properties class. This is sufficient for showing the benefits of using StringTemplate but there are some nice things that ResourceBundle does for us that the Properties class doesn’t. ResourceBundle will automatically locate the property files on the classpath and form a hierarchy of properties from most specific to most general.

If you have these property files:

Messages.properties
Messages_en.properties
Messages_en_US.properties

then

ResourceBundle bundle =
    ResourceBundle.getBundle("Messages", new Locale("en", "US");

will load them from the class path and chain them together so that if a string is missing from Messages_en_US.properties it will attempt to find it in Messages_en.properties and if it is still not found it will look in Messages.properties. This is useful so that the country file (_en_US) only needs to contain country specific translations.

Note: You may have other ways to get the locale such as request.getLocale(); in a web application.

The above code is much simpler than loading Properties objects and chaining them together yourself. The problem with ResourceBundle is that it is not readily usable by StringTemplate. Wrapping ResourceBundle in a class that implements the Map interface can solve this. I’ll show this class later but here is how it would be used with StringTemplate:

StringTemplate st = templates.getInstanceOf("templates/Test");
st.setAttribute("bundle", new STResourceBundleMapWrapper(bundle));

In templates/Test you can use bundle like so:

…<h1>$bundle.Title$</h1>…

Where the Messages* files contain:

Title=This is the Test Title

With the benefits of ResourceBundle and property files now available to StringTemplate its time to look at MessageFormat.

MessageFormat is very important for i18n because it allows translators to move substitutable text values around in a phrase or sentence to accommodate the structural differences between languages. MessageFormat has a few capabilities in its pattern syntax. Most of the time it is used in its simplest form for example:

MessageFormat.format(
	"On {0} server {1} crashed because {2}"…).

In the Web Application Internationalization and Localization in Action article there is a short example showing how StringTemplate can do what the MessageFormat {} patterns can. The example it gives is:

"title=Welcome to $username$’s test page"

At first glance it would seem that you can simply replace { and } with $ and $ in your properties files and you are done. The article then goes on to explain that this only works if the format string (Welcome to $username$’s test page) is a StringTemplate object and not a String. This can be done without having to create a named template and add it to a StringTemplateGroup. StringTemplate is smart enough to recursively process an attribute value when it’s type is StringTemplate.

There is, however, one other big difference between how MessageFormat is used and the StringTemplate example above. The title message has knowledge of another template attribute named “username”. In some cases this may be appropriate but generally:

1. the person doing the localization should not have any knowledge of the set of attribute names (the attribute names form the contract between the model and the view) and
2. a single formatted message may need to be used in multiple contexts where different attributes are substituted.

This problem is easy to fix by simply changing $username$ to $p0$ and then passing in the value for p0 in the template. For example you could do the following to define a formatted message:

StringTemplate st = templates.getInstanceOf("sometemplate");
StringTemplate message =
    new StringTemplate("On $p0$ server $p1$ crashed because $p2$);
HashMap<String, Object> bundle = new HashMap<String, Object>();
bundle.put("message", message);
st.setAttribute("bundle", bundle)
...

and then use the message from some template like so:

Error: $date, serverName, errorReason:{p0, p1, p2 | $bundle.message$}$
	

where date, serverName, and errorReason are other attributes available to the template.

The trouble is that the message should come from the message property file; it should not be hard coded. This is where the STResourceBundleMapWrapper comes in handy again. The get method can simply look at the resource bundle value and see if it has a $ in it. If it does then it must be a template so return the string wrapped in a StringTemplate, otherwise simply return the value as is.

Now this changes the rules for the translator a little. They need to be trained on the new syntax $pn$ rather than {n}. Also $ characters anywhere else in the property file will need to be escaped as “\\$”. Thankfully the complex single quote rules of MessageFormat can be forgotten.

What about the other capabilities of the message format patterns? A pattern can include format type and format style. For example {0,number, percent}. I think that the StringTemplate rendering functionality is better than what MessageFormat provides so type and style are not needed. For the most part StringTemplate rendering means that the translator has one less thing to be concerned about because the rendering is gong to do the right thing automatically.

That leaves format type choice. I don’t think this is used much but I believe StringTemplate can handle it in a similar way. The example from the SDK javadoc is:

There {0,choice,0#are no files|1#is one file|1<are {0,number,integer} files}.

To do this with StringTemplate create a properties file that contains:

DiskInfo0=There are no files.
DiskInfo1=There is one file.
DiskInfo2=There are $p0$ files.

Make it available to the template as described above through an attribute called bundle. The controller code sets attributes as follows according to how many files there are:

int numFiles = 500; // TODO get a real value
String numFilesMessage = null;
if (numFiles == 0)
    numFilesMessage = "DiskInfo0";
else if (numFiles == 1)
    numFilesMessage = "DiskInfo1";
else
    numFilesMessage = "DiskInfo2";
st.setAttribute("numFiles", numFiles);
st.setAttribute("numFilesMessage", numFilesMessage);

The template would look like this:

$numFiles:{p0 | $bundle.(numFilesMessage)$ }$

As promised, here is a sketch of the STResourceBundleMapWrapper class. This defines the constructor that takes the ResourceBundle to wrap and property methods for the same.

public class STResourceBundleMapWrapper implements Map
{
    private ResourceBundle m_wrappedBundle = null;

    public STResourceBundleMapWrapper(ResourceBundle bundle)
    {
        m_wrappedBundle = bundle;
    }
    public ResourceBundle getWrappedResourceBundle()
    {
        return m_wrappedBundle;
    }
    public void setWrappedResourceBundle(ResourceBundle bundle)
    {
       m_wrappedBundle = bundle;
    }
    …

All the optional methods (clear, put, putAll, remove) throw an exception as is shown for clear.

    public void clear()
    {
        throw new UnsupportedOperationException();
    }
    …

Because it is not intended for the map to be iterated over a trivial implementation for most of the methods will suffice. For example:

public boolean isEmpty()
    {
       //not intended to iterate over
       return false;
    }
    public Set keySet()
    {
       //not intended to iterate over
       return null;
    }
    …
	

Actually StringTemplate does use keySet when the “keys” pseudo property is referenced but it shouldn’t be needed for a resource bundle attribute. ResourceBundle does have a getKeys method but it returns an enumeration which would have to be turned into a Set. I didn’t bother.

The interesting stuff is in these two methods.

    public boolean containsKey(Object key)
    {
        if (key != null && key instanceof String)
        {
            try
            {
                m_wrappedBundle.getObject((String)key);
            }
            catch (MissingResourceException mre)
            {
                return false;
            }
            return true;
        }
        return false;
    }

    public Object get(Object key)
    {
        if (key != null && key instanceof String)
        {
            try
            {
                Object o = m_wrappedBundle.getObject((String)key);
                if (o instanceof String)
                {
                    // check for a potential template
                    String s = (String)o;
                    if (s.contains("$"))
                    {
                        // TODO cache this?
                        return new StringTemplate(s);
                    }
                }
                return o;
            }
            catch (MissingResourceException mre)
            {
                // fall through
            }
        }
        return null;
    }

If the key is not found in the ResourceBundle StringTemplate will render nothing because it first checks if the map contains the key and only looks up the value if it does. If you wanted to you could lie in the containsKey implementation and say all keys are present so that in the get method you can return the key itself if it is missing. This can make it obvious when a typo is made in key names.

Once I created STResourceBundleMapWrapper and figured out the StringTemplate syntax for passing the arguments to $pn$ attributes, internationalizing my web app was much easier.