In a couple of recent posts I have argued that a higher level language is needed to
provide a significant boost to programmer productivity in the area of web applications. This is a topic I have been thinking about for many years. I’ve worked out some aspects of what I think the language should be. A little over a year ago I even started implementing a parser for the language in ANTLR. I worked on it for a few weeks before moving on to other higher priority projects. Higher priority simply means something more interesting caught my eye. It’s finally dawning on me that I’m not going to get a big enough block of free time to work on this any time soon. I guess I prioritize the smaller projects that have a better chance of getting finished.
Instead of mothballing the project completely I’m going to share some ideas from it. This his may be of interest to others and who knows, I may get back to it from time to time.
I should start by saying that I don’t have any rigorous definition of programmer productivity or how to measure it. Since the language doesn’t yet exist there is nothing to measure anyway. The general idea is that programmers should be more productive when using an efficient language. I think of language efficiency as the ratio of meaning conveyed to words used. This is very fuzzy because I have no definition for units of meaning and even the definition of a word has some variation — do curly brackets or comments count? The general principal can be seen in human to human communication. In any highly specialized field such as law, medicine, or software, humans communicate efficiently because of a large body of shared knowledge. When communicating with someone outside the field efficiency drops off drastically because the shared knowledge cannot be assumed. Domain specific languages (DSL) have the potential to be very efficient because knowledge within their domain can be assumed. Language efficiency is an interesting topic but it is not the main point of this article. You can try this summary of an interesting paper. While the paper is not about DSLs it does try to measure programmer productivity.
As I’ve worked on web apps over the years I often get the feeling that way too much code is spent on implementation details that have nothing to do with the functionality of the app itself. Try this thought experiment: take some reasonable subset of your current application and write down just the functionality that is app specific. There will be statements about pages, what goes on each page, how pages are connected (navigation), how form controls and other page contents are connected to back end data, constraints on inputs, policies about who can do what, and style and behavior information. This is a lot of information, far more than you will find in any functional specification, but as the Uncomfortable Truths Well will tell you there is no getting around having to specify this stuff.
The question is what is the minimal amount code that needs to be written, for the language implementation to produce a functioning application? What are the assumptions that allow this to happen? This is the approach I have taken to design the language.
The language will be declarative. It does not need to be a general purpose language. It is domain specific. The domain is creating web applications. (The general principals are applicable to desktop apps as well but that is not my current focus.) It is declarative in the sense that the code specifies what must be done but not exactly how. In this regard it is more like SQL than Prolog (Prolog is general purpose).
Object oriented programming and procedural programming have little to offer at this level. If you went through the thought experiment and came up with things like; “the page will be represented by an object with these methods….” you need to take a step up. The bulk of what a web app does is transfer data from a database (or other data store) to form fields so a user can view and change it and then save the data back to the database. The procedure for doing this is mostly repetitive and should be abstracted away. Writing code to move data in and out of objects is not productive.
There are many examples of declarative languages in the web space including: HTML, and CSS. (XUL is a similar example for desktop apps). These show the expressiveness, power and benefits of declarative programming but none are all encompassing or extensible. They cover aspects of web application specification but not the whole specification and they are not designed to be extended so they can do so.
The language will span the whole application from the UI, including look and feel, to the data access layer. It probably won’t include the data source implementation except possibly to describe it in enough detail for the data access layer to be declarative. If the data store is a SQL database the source code might include SQL statements or enough detail for SQL statements to be generated but it wouldn’t include stored procedure code.
There are a number of other requirements that get taken for granted. Things you want and expect in your app but didn’t specify because they should apply to all apps. They should be secure (for example, free of XSS and CSRF holes), accessible, scalable, and responsive. These things should be built into the language implementation. Where trade offs are necessary for example perhaps between accessibility and responsiveness implementation parameters (think compiler switches) should be used rather than requiring source changes.
Many current frameworks have limited support for the above assumed requirements. Because they work within a general purpose language there are usually ways to meet the requirements but they are not automatic. A pure server side or pure client side framework is limited in the amount of optimization it can do. For example a server side framework that requires lots of session state is going to have issues with scalability. To get around this it may also provide or rely on complicated session replication.
The language could be interpreted or compiled or some combination of both. Compiled most likely means code generation to another high level language rather than direct to machine code (virtual or physical). Either way the implementation would be based on some combination of languages and frameworks available today. It is possible for there to be multiple code generators or interpretors such that different combinations of languages and frameworks could be supported without having to make source changes. For example the same source could run on a Dojo – Java – Struts – JSP stack or a jQuery – PHP stack. The extent to which source changes are required depends on the similarity of the two implementations. Clearly this is a lot of infrastructure work and may or may not be worth it but the main point is that the specification of the application is independent of these implementation details.
Some wisdom can be borrowed from HTML. The key thing about HTML is that it provides a literal syntax for the tree that represents a page. This is also true of other user interface markup languages and JavaFX as well. The problems with HTML from an application specification point of view are:
- The focus is on a page rather than a whole app. This makes it impossible (within HTML) to factor out common parts and do global optimizations.
- It is not extensible. The semantics of the nodes of this tree are fixed and built into the browsers.
- It does not support composition. The whole page structure must be in a single file (ignoring frames).
- No separation between data and controls.
- The syntax is better suited to text markup than tree building.
The language will allow the creation of a single tree that describes the whole application. The nodes of the tree define the semantics of that part of the app. The nodes and their properties convey all aspects of the application including style (look and feel), behavior, navigation, validation, model schema, constraints etc. A node could represent a standard HTML text input, a tab control that contains a number of tabs or a full featured grid control. Nodes have properties that define aspects of the node.
Clearly a tree that describes an application, even a small one, in detail is too unwieldy to be given in a single file or even visualized all at once. The tree is made up of fragments and stitched together by references between fragments. This is similar in concept to the way RELAX NG schema builds a single tree from named patterns, references and external references.
A single file can contain one or more tree fragments and the whole tree can span multiple files. One node is designated the root. Tree fragments can be parameterized, can be composed of other fragments and referenced multiple times. New node types can be defined that extend other nodes. Named constants and property sets can also be defined and referenced.
The language can be divided into two layers. The first is the general tree construction capabilities that are independent of what the tree nodes are. The second layer is made up of an implementation defined library of node types which define the range of possibilities for the application. Nodes represent UI components and schema elements of the application data model. It must be possible to define new node types and provide implementations in order to extend the range of what an application can do.
The syntax of the language is not XML. The details of syntax are beyond the scope of this article. In my incomplete prototype I borrowed ideas from Python and YAML. Later I learned about Java FX and I really like its literal syntax for UI trees, which is somewhat similar to JavaScript or JSON literal syntax for data. The main point is that it should be easy for humans to read and write. It should not be overly verbose.
Even more wisdom comes from CSS. The function of CSS is similar to aspect oriented programming (AOP). The join-points are the style properties and content of elements, the point cuts are the CSS selectors, and the advice is a set of style properties and values. This is probably stretching the definition of AOP but the point is that CSS provides an efficient way to specify cross cutting concerns of application style. CSS applies at the level of HTML and is restricted to a fixed set of styles. There is no reason that the concepts of CSS should be restricted just to style. For example, some people have used the syntax of CSS to declare behaviors.
The language provides a way to inject new property values, node content and even new nodes into the tree. The syntax may not be the same as CSS but the concept is. The nodes of the completely assembled tree are addressable by type, id, class (and probably other characteristics). The CSS like tree modifications are expressed as a set of rules (aspects) consisting of a selector (point cut) and a list of node modifications (advice) including property values, node content and node fragment insertions or node removals.
The benefits of this CSS like aspect oriented declarative programming are separation of concerns, flexibility, and application extensibility.
CSS provides a great benefit in the separation of style from markup. The implementation of the application will naturally use CSS internally. The draw back to CSS is that the styles are fixed. There are many things that are really stylistic but must be put in the HTML markup. For example: placing labels above, to the right or to the left of a control. Another example: making a single selection from a list could be done with a select element or a group of radio buttons.
Because the language allows the definition of higher level controls it only makes sense that style be applied to those higher level controls and not to the HTML or DOM nodes that end up being generated from the source. There is no need to edit CSS as a source file. It will be generated based on style information in the application definition tree. The aspect orientated feature of the language is used to separate the style concern from other concerns. The style information is kept in a separate file. This same language feature can be used for more than just style. Another file could have rules defining UI behaviors.
This same feature enables an application to be extended by third parties or even end users (assuming you allow them to add new aspect files) without them having to modify any source files. They just need to know the point cuts at which to insert the new functionality. It could be anything from adding a text input field to a UI page along with a new property to the data model, to inserting whole new sets of pages and data models. (The term page is used loosely here and really refers to some region of content in the UI.)
Taken together, the tree building from fragments and the CSS like aspect oriented tree modifications make for a very flexible system for defining an application. The intent is that developers should have a great deal of control over how they modularize the code and what concerns are separated.
Another coding burden is internationalization of text. The text translation calls are just repetitive noise. In Java the code often looks like: String label = mybundle.getString(“OK.button.key”); when it would be much clearer to just have String label = “OK”;. Tools have been created to help. For example some editors will find string literals and replace them with text translation calls and include comments in the code to keep track of the work they have done. All this would be much simpler if it were built into the language. Most string literals in an application do need to be localized.
In the language all strings are internationalized by default. It takes slightly more effort (perhaps a character or two) to write a non-i18n string literal. Strings use the key-less method like gettext to remove the burden of having to think up keys and keep them in sync between source code and localization files. This abstraction allows the language implementation to determine the best place to do the localization, which could be on the server, for example using Java resource bundles, on the client using JSON or even localizing static HTML resources.
The language has an underlying data model that is probably similar to JavaScript. It has objects/hashes, arrays/lists and scalars. The data model is separate from the UI presentation. Binding expressions establish linkage between parts of the data model and properties of UI controls. When the control property value changes so does the bound data model value and when the data model value changes so do any bound UI control properties. These concepts of separation of data model and UI presentaton and data binding can be found in XForms and I believe in JavaFX as well as others. The bindings declare a relationship between the UI and data and specify how and when data flows between them. A similar kind of binding establishes a relationship between the data model and the back end data store. Bindings eliminate a lot of tedious code that moves data around in response to various events.
One concern you probably have is how can all the logic of an application be specified in a declarative language, especially one that isn’t general purpose. In other words how is arbitrary computation done? The answer is that it is done outside of this language. It is done in a traditional language and exposed to the declarative language through node types and functions that can be called from binding or value expressions. The built in set of node types and functions should handle the common needs of applications. For special cases you can extend the capabilities of the language by creating new nodes or functions. For example: if you need a new UI widget create a new node, if you need a special validation write a function for it, if you need to connect to a different kind of data store this would probably involve a few new nodes.
The implementation source code should be customizable so that it can change to meet the needs of each application. Another place where application logic can live is in the data store. If the data source is a database the logic could be in stored procedures. If the data store is a RESTful web service the logic can live behind the resources. If the data source is a bunch of Java objects then logic can live in them.
This establishes a separation of concerns between the specification of the application and the implementation. One group of people can work in the declarative language and another (probably smaller) group can work on the language implementation extensions.
I believe a language like this would be of most benefit in these cases:
- Large applications. These usually have a lot of repeated patterns and common set of UI controls and are connected to a single data store.
- Implementing a number of similar applications over time. Initial effort in creating the infrastructure would be amortized over a number of apps.
- Applications that need to be customized for each customer. The declarative source should be much smaller than the equivalent functionality in any other language and framework. Less code means it can be changed faster and with fewer mistakes. This is the use case that originally got me thinking about these issues four years ago.
- Applications that need to allow customer or end user customizations.