Category Archives: StringTemplate

StringTemplate for JavaScript

About three weeks ago I announced on the StringTemplate discussion forum that I had updated my StringTemplate Standalone Tool (STST) to work with StringTemplate version 4. It has been a number of years since I have used StringTempalte and almost as many since I’ve programmed in Java. So why this return to StringTemplate and Java? My motivation for updating STST is that I intend to create a pure JavaScript implementation of StringTemplate.

So far my work with JavaScript has mostly been creating UI widgets as part of existing web apps and frameworks. I have not had a need for templates, at least not ones processed by JavaScript. That may change with my Climbing Comp project and some other code generation projects I have in mind.

What I’m looking for in a JavaScript template engine:

  • Strong separation of presentation and logic
  • Compiled templates, good performance
  • Mostly need to run on the server using node.js but running in the browser is a plus
  • Can generate any text format, not just HTML

The best match I found is Handlebars. It compiles to JavaScript, runs in the browser and node.js environments, and does a decent job of keeping logic out of the templates. It seems focused on HTML generation but I think it could work for other text formats. I have only played with it a little. There are a few things that I don’t like. Having to write helpers in JavaScript and the way partials are handled distinct from templates doesn’t seem as nice as StringTemplate. Handlebars doesn’t have the separate renderer and writer layers that StringTemplate has. I didn’t do a detailed analysis of the differences between Handlebars and StringTemplate. My gut tells me that I would prefer StringTemplate.

Because, at least initially, I plan to do generation off-line on the server side I could use the Java version of StringTemplate but I would prefer not to add Java to the tool chain; so far I have been able to do everything I need with node. This is what has brought me to creating StringTemplate for JavaScript.

StringTemplate for JavaScript (StringTemplatet-js) will compile templates to JavaScript. It will not have an interpreter. It should be 100% compatible with StringTemplate v4 template syntax and semantics. The API will be different.

To implement StringTemplate-js will require parsing template files and generating JavaScript code. The former would be a good job for ANTLR and the later well suited to StringTemplate. ANTLR can generate JavaScript code but again I would rather not have to use Java. This lead me to try two JavaScript parser generators: jison and pegjs.

I tried jison first and it was a failure. Any syntax error in the grammar resulted in a generic exception with no indication of what went wrong or even the line number. I found my errors by selectively commenting out code until the problem line was isolated. The generated code was extremely difficult to understand and debug. Then when I tried examples right out of the documentation that didn’t work I gave up.

Now I’m using pegjs and it is working very well. The error reporting is very good and the parser it generates also has very good error reporting including source line and column. One minor issue is that it stops at the first error. The generated code is not as readable as what ANTLR3 generates but it is reasonable and I have had no problem debugging it. Not having a separate lexer and parser takes some getting used to. It means having to represent white space throughout the grammar because you can’t just toss the tokens out in the lexer. Some of the things I thought would be difficult, such as having configurable start and stop template delimiters, turned out to be quite easy to deal with.

StringTemplate-js will use itself to generate the compiled template JavaScript code. This is where STST comes in. I plan to use it to bootstrap the process. It will also come in handy in validating that StringTemplate-js gets the same answer as the Java reference implementation.

Hopefully soon JavaScript will have another template engine option. Watch this space.

Script injection and StringTemplate

I just read a very interesting article called Secure String Interpolation in JS that got me thinking about rendering in StringTemplate. I don’t follow the caja project but it looks interesting. I found the article linked from John Resig’s blog.

Script injection in web applications, which also goes by the name XSS (Cross Site Scripting), is a serious and common security problem. It arises when the application does not appropriately escape data that comes from an untrusted source. The data could contain malicious JavaScript that would get executed. If developers are burdened with remembering to escape each and every instance of data they will eventually forget or make a mistake and a security hole will result. One main point of the article is that string interpolation should be automatically secure — it should do the right escaping for you.

Every now and again the issue of how to escape special characters when generating HTML files comes up on the StringTemplate mailing list. The answer is very straight forward – write a renderer. My main point, which I’ll get to eventually, is about the details of the renderer.

The author of the above article distinguishes string interpolation from “full blown” templating languages. JSP, PHP, and XSL were given as examples. Three problems with full blown templating languages were given (folks familiar with StringTemplate know there are more):

  1. They do nothing to help solve the escaping problem.
  2. They are verbose and add lots of boiler plate.
  3. They don’t make simple things simple.

I think StringTemplate holds up well against these complaints. The renders can do escaping. The simple templates (.st files) have no boiler plate at all and the group files (.stg) add only a little. Lastly, it doesn’t get much simpler than “Hello $name$!”.

The most important issue and the one the article is really about is number one — escaping.

On an HTML page what characters need to be escaped depends on the context: element content, attribute value, JavaScript, and URIs. StringTemplate rendering assumes a single rendering context. As of StringTemplate version 3.1 the format option can be used to specify a specific rendering (format). The trouble with this is that the template writer must be aware of the different contexts and what format to use in each. An alternative is to come up with a one size fits all escaping renderer. This will end up escaping more than it needs to. There may be some fringe cases where this could cause problems.

The thing I found most interesting about the approach given in the article is that the context scanner keeps track of the current context in the output so that the escaper can do the right escaping according to the context. To put this in StringTemplate terms the output after rendering would be feed into a parser that tracks changes in context (element to attribute value to JavaScript to element content etc.) and informs the renderer(s) of the current context.

You can still use the format option or other custom renderers to specify how the data is turned into a string (for example a number could be currency or a percentage). This is something that the template author is expected to know. It seems reasonable to me that it would be better if the author didn’t need to understand the intricate escaping rules of web pages.

A key innovation of StringTemplate is the identification of the Model-View-Controller-Renderer pattern — that there is a renderer distinct from the view. (Read about MVCR here.) Now I am seeing the render as two distinct pieces. There is the format renderer and the escaping renderer. The former is concerned with if negative 23 should be output as (23) or -23. The latter is concerned with if single quote should be output as ‘ or ' The format renderer runs first and passes its output to the escaping renderer.

The reason it makes sense to think of two different renderers is that they each have different masters. The format renderer is directed by the needs of the application, the locale and perhaps presentation directed formatting with the format option. The escaping renderer’s function is determined by the language of the output: SQL, HTML, XML or just the text of an email message.

Other thoughts and observations:

Is there a need for a JavaScript implementation of StringTemplate? As it becomes more common to move the model, view and controller to the client there could be a role for it. It could also run server side in Rhino. Although the author of the article states that it is not a goal to create a template language I wonder if the extra capabilities of StringTemplate would be welcome.

The method used in the article for implementing auditable exemptions (cases where you know the data is safe and you don’t want it escaped) is the same as would be done in StringTemplate. Just wrap the data in a type that returns the raw string when rendered.

In the article a simple FSM was used to parse the HTML. Only an excerpt of the code was shown and I didn’t bother to download it and look deeper. It seems to me that the output HTML needs to be well formed otherwise the parser would be very complicated if it has to deal with the tag soup that browsers deal with.

This is now at least the fourth time I’ve seen a parser written in JavaScript so I think there could be a need for a JavaScript backend for ANTLR. I would have liked to create a l-system language parser in JavaScript for a recent project of mine. Is there already an ANTLR for JavaScript that I don’t know about?

Does anyone else think that it would be useful/feasible to have a HTML escaping renderer (that does the correct escaping in automatically) as part of the StringTemplate library?

L-systems in JavaScript using Canvas

While playing with Scratch I was thinking about what kinds of things would make good Scratch projects. I have had an interest in L-systems for a long time now but never did more than read about them and look at the pretty pictures. L-system tree I thought scratch might motivate me to do something in this area but I quickly realized with Scratch’s lack of data structures (besides the 2d drawing surface) and argument passing it would take all the fun out of doing an L-system implementation. Sure, people have come up with ways of drawing fractals in Scratch but these are fixed algorithms usually at a specific generation. I have not yet seen an L-system implementation in Scratch.

About a month later I went to the ajax experience conference and learned about the new HTML canvas element among other things. I wanted a project to get practice writing JavaScript, more experience with using jQuery, and to try out jQueryUI and canvas. L-systems seemed like a great project for this purpose.

Before I get to my project here is a real quick definition of what an L-System is. A Lindenmayer system (L-system for short), named after biologist Aristid Lindenmeyer, is a rewriting system. It takes a list of symbols and repeatedly generates a new list of symbols by following a set of rules. Each new list of symbols is called a generation. The symbols can be given a graphical representation to produce an image algorithmically. The beauty of L-systems is that with just a few symbols and rules amazingly complex images can be created. L-systems were developed to model plants but can also draw fractals. More information is available from the links above and from the lsystem.js documentation.

My project, L-system.html, is a serverless web application for exploring a number of plant and fractal models created with L-systems. You select a model, change its parameters such as angle or distance depending on the model, set the number of generations then press the play button to see it drawn.

Being able to adjust the parameters of a model helps to understand it. When you see a fractal of some reasonably high number of generations it is not always clear what process gives rise to it. Stepping through from the zeroth to first and second generation of a fractal really helps to visualize the rule that makes the fractal what it is.

L-system.html is not very fancy or even complete. The main purpose for me was learning canvas, jQuery, and jQueryUI. It makes use of two reusable JavaScript libraries that I wrote:

  • lsystem.js – a L-system library that supports parametric or non-parametric, deterministic or non-deterministic, context sensitive or context free, and branching or flat models.
  • turtle.js – A 2d turtle graphics library on top of the HTML canvas API.

Both of these are open source with a BSD license if you care to use them. They can be used together or independently. You can download the whole project. If you make improvements to them or have suggestions I’d love to hear about it.

The rest of this post covers what I discovered along the way.

First I have to say I’m impressed with canvas and its capabilities. Its not as full featured as most mature graphics libraries and there are a few features that are not supported on all browsers but it opens up another area to web applications (without having to resort to Flash) that was previously only found in desktop applications. People are doing cool things with it already. Support for IE is accomplished with explorercanvas from Google. I have tested my project on Firefox 2, IE 7, Opera 9.24 and Safari 3beta (on Windows) and it works well except that the save functionality doesn’t work in Safari or IE. Also I have noticed IE to be slower than the others but have not done any rigorous timing.

Thanks to Canvas Paint (a canvas implementation of Microsoft Paint) for the technique for saving canvas images.

L-system.html is an example of a serverless web application. Its an application because it is interactive and does something at least a little interesting or useful. Its a web application because it runs in a web browser. Serverless is a somewhat new category of web application. It basically means that none of the application behavior is running on a server. A web server may be serving the HTML and related resources but there is no servlet, or PHP script, etc. A server may be providing raw data or persisting data (like the Amazon S3 service) but this is just data storage or acquisition and often the servers or services are not even owned by the creator of the application. In some cases the application can just be an HTML file on your local file system. L-system.html can run this way. An even cooler example of the kind of interesting things that can be done without a server is TiddlyWiki – a serverless wiki.

After working more with jQuery I’m convinced that this is the way I want to write JavaScript and to work with the DOM. However I’m not anywhere near as impressed with jQueryUI. It just doesn’t seem ready for prime time. I only wanted a few UI elements: a slider, a dialog, a progress control and a color picker. There is no progress control or color picker in jQueryUI although there are plugins for color pickers and maybe a progress control as well. I thought it would be best to stick to the official UI widgets. I struggled to learn the UI widgets and get them to work the way I wanted. I needed to read the source as well as the documentation to figure them out.

The first problem I ran into was with the slider on on IE 6 and 7. I found a bug report that said the problem was fixed in the latest. After downloading the latest from the SVN source repository the problem was indeed fixed.

I wanted to be able to move the slider with the right and left arrow keys and to show that the slider had focus. I know that jQueryUI doesn’t have full accessibility support yet, but keyboard support seemed like it should work for a control like slider. I noticed some code to turn on keyboard support when webforms (not sure what that is) is present. This helped me figure out how to get it working the way I wanted. You can tab to or click on the slider to give it focus and then use the arrow keys to move the slider.

One thing that bugs me about some ajax apps is that they break the back button. I really wanted to get back button history working with jQuery in this project. At the ajax experience conference I saw the history plugin demonstrated and it looked so easy. So far my attempts to get it working have failed. The history plugin seems like it would be easy to use with the tab control but i’m not using the tab control. Then I read that the new tab control in jQueryUI doesn’t work with the history plugin. There is a mention that the history plugin needs to be rewritten but no time frame for when it will be available. Since I’m not using tabs it may be possible to get the current history plugin working but I haven’t figured it out yet.

My next problem was with the dialog control. I found that I could end up with multiple copies of the dialog displayed. I decided to try the latest version from SVN like I did with the slider. I did manage to solve my problem by using the new dialogInit method. (There may have been another way.) The trouble is that at the time I got the latest version of dialog the mouse management was rewritten which broke the slider and resizing of dialogs. As of this writing they still are not fixed. There may be some set of source versions that will solve both my slider and dialog problems but I don’t have time to find it. So at this point the sliders do not move with the mouse (you get a JavaScript error). I’ll just have to wait for the jQueryUI mouse implementation to get finished.

I noticed that some of the jQueryUI documentation (at least the examples) are in advance of the current 1.0 release. For example, the dialogInit method was shown in one of the dialog examples even thought it is not in the 1.0 release.

There is no progress control in the jQueryUI so I made my own. The next step will be to learn how to turn it into a plugin.

Along the way I found a number of useful tools and practices for writing JavaScript

  • Firebug is a very good tool. I knew that already but with this project I used the profiler for the first time. The profiler is very easy to use and helped me to improve the performance of the L-system algorithm. I also used the console log, which Firebug lite makes work cross browser.
  • I’m using JSLint from an ant task and it has found some problems. I described the technique for ant integration here.
  • I used jsdoc-toolkit to generate the library documentation. This tool is very easy to use and produces nice results. Since the JavaScript is going to get minified anyway I think it is a good idea to keep the documentation in the source.
  • For testing the 2 library modules I used jsunit. Actually I only used the assertion functions from the core. This looks like it could be a good tool but I need to learn more about how to use it effectively.
  • I use just the CSS library from YUI.
  • To build the final html as well as the aggregated and minified resources I use ant, yui compressor, and STST. One thing I noticed is that yui compressor catches some JavaScript problems that JSLint did not.

After I was just about done with this project I discovered that someone else created a canvas L-system implementation LSys/JS. Although it doesn’t support as many kinds of L-systems it does allow you to enter your own. Also it does not have a reusable library layer.

If you have read this far without trying it out you should take a look at L-system.html now. If you want to add your own L-system models you can download the project and modify the lsystem.html file. If you come up with a cool model leave a comment with a link to it.

Thoughts on StringTemplate part 1

I have been working with StringTemplate for a few months now. I know that isn’t too long and others have been using it for much longer. But in that time I have thought a lot about it and formed some opinions. (I hope it is clear from what I have already written that I’m a big fan.) In this and the next few articles I’ll try to organize my thoughts, hopefully stir up some discussion and possibly impact future versions of the language.
Continue reading Thoughts on StringTemplate part 1

On Learning StringTemplate

I have been thinking lately about the learnability (the ease with which something can be learned) of StringTemplate. I like StringTemplate and would like to see more people make use of it. To that end I think there are some things that can help make it easier for people to get started with it.

StringTemplate is both a language for describing text generation and a tool (template engine) that implements that language and provides an API to connect the tool to another program.

StringTemplate is not a general purpose language and cannot stand on its own. By design it has no way to perform general computation. It also has no general way to create data and limited ability to specify literal data. It is up to some driving program to produce and or make data available to the StringTemplate template engine. This is not a negative – this is it’s strength. See this excellent paper for why.

However, the lack of readily available data makes learning the template language difficult without also learning the StringTemplate API. In order to try StringTemplate out one must learn the template syntax and semantics, and the API and then write a program to produce some data and call the template engine.

In theory one benefit of the separation of presentation from business logic is that a project can be divided between people who work on the business logic (the program) and people who work on the presentation (the templates). The people working on presentation don’t need to be programmers.

The current documentation for StringTemplate describes both the template syntax and the Java API at the same time. In some cases it is confusing to learn two things at once and more importantly it excludes non-programmers from learning StringTemplate.

In contrast, trying out XSLT is is easy (even if mastering XSLT is hard) because all you need is a text editor to create both your input documents and your style sheet and an XSLT program to process the style sheet with the input documents. Likewise trying out HTML is easy, all you need is a text editor and a browser. Interpreted languages like Python that have an interactive mode also make it easy to jump right in and get the feel of things. (I’m not saying that StringTemplate is like XSLT, HTML or Python, I’m just comparing the ease with which one can dive in and try the technology out.)

So what StringTemplate needs is a literal representation for data that can be used as input and a stand alone program to read the data and templates to produce the output text.

I choose JSON as the literal data format. The reasons are:

  • Its a standard
  • It is becoming increasingly well known
  • It is easy to type in and not overly verbose
  • I had already made changes to get a Java JSON library to play nice with StringTemplate.
  • It supports all the data shapes that StringTemplate works with.

I created a command line tool called StringTemplate Standalone Tool (STST). It takes as input the name of a file containing data in JSON format, the name of a template and optionally the name of a group as well as various options.

Look ma, no programming.

Here is a simple example of how it is used.

Create a JSON file called data.js with this content:
{ "audience": "World" }

Create a StringTemplate file called first.st with this content:
Hello $audience$!

From a command prompt in the same folder as the above two files type
> stst first data.js
The output is
Hello World!

It also works with group files and can use the angle bracket lexer. For command usage type: stst -h

The next thing StringTemplate needs is a tutorial that focuses on the template syntax independent of the API. STST enables this kind of tutorial. Separate documentation can focus primarily on the API.

Download STST here

This tool can have other uses as well. Here are some ideas:

  • It is a quick way to try out template syntax without having to write or modify a program. This can benefit seasoned template authors as well as beginners.
  • It could be part of a test framework for StringTemplate itself or for a set of templates in an application. The templates can easily be processed with known data and the expected results asserted.
  • It enables development of the templates before or in parallel with code by creating a JSON data file that represents what the code will produce.
  • It could be used as part of shell scripts where some existing program outputs JSON data and STST is used to generate text from it. The data could come from a web service for example.

What could you use it for?

Future possible directions for STST

  • Integrate the functionality into an IDE for StringTemplate
  • Expose the functionality in an interactive servlet (web app).
  • Support other data formats like CSV, YAML, or XML
  • More support for renderers