Thoughts on StringTemplate part 2

Part one in this series mainly focused on the data model of the StringTemplate language. In this post I’ll talk about programming-in-the-large with StringTemplate.

Programming-in-the-large is about how to deal with creating and maintaining large programs. Of particular interest are the facilities a language provides for modularization and how the modules interact. These interactions may touch on issues of information hiding and the abstractions available in the language.

StringTemplate allows you to organize your templates in different files. There are two types of template files: the simple template file and the group file. The simple template file is often just called a template although this can be ambiguous because groups contain templates and templates can even be created programmatically. I use the term simple template file after the XSLT term simplified stylesheet module which is a stylesheet that starts with a literal result element.

Simple templates are just files with StringTemplate actions between delimiters ($ $ or < >). There is one template per file. They cannot declare arguments, the name of the template is the name of the file (less the extension) and the extension must be .st. Simple templates can invoke (also known as call, reference or include) other simple templates by using a relative path to the other file. For example:

/dir1/template1.st:
  This is the caller. $dir2/template2.st()$
/dir2/template2.st:
  This is the callee.

Group files allow you to define several templates in a single file. The group file must have a .stg extension. Group files also support inheritance, and interfaces (more on interfaces later). The templates in a group file can declare arguments. Templates can invoke other templates in the group file or any of its super groups. For example:

supergroup.stg:
  group supergroup;
  util(arg1) ::= <<
  This is in the super group. $arg1$.
  >>
subgroup.stg:
  group subgroup : supergroup;
  template1(arg1) ::= <<
  This is in the sub group. 
  $util(arg1)$.
  >>

Now here is where implementation details get in the way. For simple templates to invoke other simple templates they must be gathered together in a StringTemplateGroup object. This object is given a root directory and all template files must be under that root. This same object when constructed differently represents a group file. Templates in one group cannot call templates in another group unless the groups are in the same inheritance hierarchy. This means that templates in a group cannot call simple templates and vice versa. (Technically it should be possible to programmatically cause simple template groups and group file groups to inherit from one another but it can’t be done from within the group or simple template file syntax.)

I have been using StringTemplate to generate the HTML for a web application. I started out using simple templates but soon found I wanted the ability to declare arguments on my templates so I switched to the group file format.

The program calls a template that will produce a complete web page. That template relies on other templates to generate parts of the page. These other templates in effect represent controls — aggregations of HTML such as an input field and its label, an image with a drop shadow, or a table with next and previous links. I use one group file for each page. The controls are reusable so they shouldn’t go in the group for a specific page. The only solution available is to put them all in a base group and have the page groups inherit from that base group.

There are a number of problems with this. The first is that all the control templates must be put together in a single group (because a group can only inherit from one base group). This group file quickly gets large. Second it forces an is-a relationship where there isn’t one. If my base group is called “dumping-ground-for-all-controls.stg” and my page group is called “accounts-page.stg” it doesn’t mean that an accounts-page is a kind of dumping ground for all controls as inheritance would imply.

I should be able to choose my own way of organizing my templates into groups. I would much rather put each control in its own group or possibly group like controls together. My page group would then call templates in any of these group files.

I wouldn’t use inheritance unless one page or control was truly a specialization of another page or control.

To get a better understanding of the StringTemplate notions of templates and groups it is useful to compare them with the abstractions of other languages.

Conceptually a template is like a function. It takes a collection of named attributes and returns a text string. Simple template files and group files are modules. They are the building blocks of a StringTemplate text generation system.

Languages like C allow you to group your functions together into files (modules) any way you like and functions can call functions in other modules. StringTemplate should allow the same.

On the surface group files may seem like classes. They have some things in common but also important differences. A group is a collection of related templates in the same way a class is a collection of related methods. The thing that relates a class’s methods is access to and operations on a common set of data. This data is an object or class instance. In StringTemplate there is no mutable state so it makes no sense to speak of class instances or objects. Groups support inheritance like classes do but template polymorphism is a little different from traditional OO languages. In OO languages like Java or C++ the method called depends on the type of the object the call is made on rather than the type of the object reference. In StringTemplate, again, there are no objects so the template called depends on the group used to make the initial template call. It is a good thing that the StringTemplate group abstraction was not called a class lest someone take the analogy too far.

StringTemplate groups do a good job at providing needed capabilities for programming in the large. Specifically grouping like templates together in a single file for convenience, declaring template arguments, inheritance and template polymorphism, and interfaces. They fall short in assuming that all templates needed by a text generation system belong in a single group hierarchy.

It may seem restrictive for simple templates to limit the file to containing a single template. Imagine C forcing you to have one function per file. One might be tempted to get rid of the simple template and just keep the group file. However there is value in having simple templates in addition to groups. The simple template is useful when the output is mostly static text with just a few holes that need to be filled in with data. The simple template is like a form letter (Dear $name$, You may have already won a million dollars…). These templates are the easiest for non-programmers to understand because there is no extra syntax beyond the template actions and the template looks very close to what the final result will be. The same argument for the existence of simplified stylesheet modules in XSLT can be made for simple templates.

I said I would come back to group interfaces and this seems like a good time to do it. I think StringTemplate gets interfaces exactly right. An interface provides a contract between the caller of templates (the program) and the templates themselves. The interface defines what templates the program plans to call and that the group must implement.

Interfaces in StringTemplate work very similar to Java interfaces. The interface specifies a set of template names and their arguments. When a group implements one or more interfaces it is agreeing to implement all the templates from the interfaces with the right number of arguments for each. An error is given if the group doesn’t implement the templates specified in the interfaces. If the writer of the program changes the interface then the writers of the group files will know that they need to make corresponding changes.

Another difference between groups and simple templates is in their ability to reference other groups or templates in different folders. Simple templates can use relative paths to reference templates in other folders. Groups can not use relative paths to reference super groups, or interfaces. However the implementation allows specifying multiple directories to search for super groups and interfaces but simple templates must be under a single root. I view these as implementation details and not limitations that are inherent in the language.

To better support programming in the large I would like to see StringTemplate meet these requirements:

• allow multiple groups during template processing
• allow mixing simple templates and groups. A simple template can call a template in a group and a template in a group can call a simple template.
• support some kind of namespaces for simple template, group and template names

Here is a proposal for meeting the above requirements. Groups and simple template files, collectively modules, are kept in a multi-rooted hierarchical store such as a file system. Other stores such as a database or compressed archive could be used as well — even a mix of them. The StringTemplate engine implementation would provide a way to specify the roots where modules are found. Something along the lines of the Java class path would be one possibility. The implementation could borrow some ideas from the Java ClassLoader. I’ll call the hierarchical containers of modules folders. They could just as well be called packages or directories.

Example:

controls/listview.stg:
    listView(id, caption, data) ::= ...
    listCell(item) ::= ...
    ...
controls/form.stg:
    button(id, value, label, accessKey) ::= ...
    checkbox(id, value, label, accessKey, checked) ::= ...
    ...
pages/users/list.st
pages/users/add.st

The above example defines two groups (listview and form) and six templates (listView, listCell, button, checkbox, list, and add) in two folders (controls and pages).

A group file would introduce a kind of virtual sub folder that contains templates just as a normal folder contains simple templates. The fully qualified name of a template is the forward slash separated list of folders, possibly including a trailing group name and finally the template name. For example the template pages/users/add would add a button to the page by referencing $controls/form/button(...)$ and template pages/users/list would reference the listView control as $controls/listview/listView(...)$. The listView template would reference the listCell template simply as $listCell(...)$. The general rule is that templates in the same folder or group don’t need any folder or group path in front of it. If part of a listView control included buttons then the listView template would refer to the button template as $controls/form/button(...)$.

The hierarchy of folders provides a kind of namespace similar to Java packages. Folder, simple template, and group names are in the same namespace. You can’t have a folder, group, or template with the same name in the same folder. Even though the file system will allow a folder to contain sub folder foo, and files foo.st and foo.stg this would not be allowed. Folders can contain sub folders, groups and templates. I seen no need for groups to contain other groups (as in Java’s inner classes).

I used the ‘/’ character rather than ‘.’ to separate folders, groups and templates because ‘.’ is already used to specify nested attribute data and because it is already used in simple templates.

The syntax of group files and even simple templates should be extended to support importing folders or groups so that the templates in those groups and folders can be used without having to use fully qualified names. Different languages have different functionality for one module specifying other modules it intends to use. In StringTemplate I think it should be possible to specify a specific template in a group or folder or all of them. It should be possible to provide an alias for a template.

As for the syntax I’m not sure what to do. It might look something like this:

• $use controls/listview/*$ to import all the templates from group listview.
• $use controls/listview/listView$ to import just the listView template from group listview.
• $use controls/listview/listView as list$ to import the listView template but call it as $list(…)$

In a group file the use directive could go at module scope without being enclosed in the action delimiters. At module scope it would apply to all templates in the module. For example:

  group foo;
  use bar/template1;
  ...

Another important principal in OO languages (and even languages like ADA) is information hiding. Classes can declare their members, data or methods, as public, private or protected. Since template groups have no mutable state there is no data to be hidden.

I think it may be useful to specify templates as being private. Some templates are just a useful way of implementing another template and don’t need to be called from outside the group. The implementation of one template can be broken down in to a number of sub templates. The details of which are not important to the callers of the main template. The main template is public while the sub templates should be private.

One possibility would be to add keywords like public or private to template definitions in a group file but there is no elegant way to do the same for simple templates. I think it would be fine to use a naming convention such as template names beginning with an underscore can only be used from templates in the same folder or group.

I believe these enhancement will make reusable template libraries possible. Even if there isn’t much need for template libraries a large project will benefit from being able to organize the templates in a way that makes sense for the problem at hand. You can choose to use simple templates or groups depending on what is best for each template rather than based on rules for what calls are allowed.