APIs, UIs, and REST

When you read about REST these days it is usually in the context of APIs. You’ll even find REST being discussed in Web Service and SOA circles. A REST API makes an application’s data and functionality available to other programs according to the principal of REST.

Sometimes you see the term REST API and sometime REST Service (or RESTful Web Service). I don’t like the REST Service term because I think it gets confused with the SOAP and RPC styles. REST API is also a strange term. When we think of APIs the most familiar is the local call to a function, method, subroutine or procedure. The remote procedure call tries the best it can to preserve the same semantics when the caller and callee are not on the same machine. When we think of a remote API it is easy to assume RPC. With REST the focus is on the resources. Other aspects of the interface such as the set of method are fixed and uniform. I use REST API to mean a RESTful application data layer that can be consumed by another program.

REST is an architectural style for hypermedia systems with the World Wide Web being the primary example, and the only instance of immediate concern. For all the talk of APIs the web is still mainly about UI. Its a hypermedia system for humans.

REST doesn’t care if it is a human or a program that is the client of your resources. So why all the focus on APIs? Other than general information about REST I have not found practical specific advice for implementing RESTful web applications. The best I have found is this slide deck — I wish I saw the talk.

An information only web site (one with static HTML and no forms) is likely to be RESTful even if only by accident. As web sites allow richer interaction between the user and the site and among users of the site it gets easier to stray from the principals of REST. Some of the issues that come up are:

  • The HTML form element doesn’t support PUT and DELETE
  • You need somewhere to put conversational (i.e. session) state and REST says no cookies
  • You need to authenticate users but can’t accept the limitations of HTTP authentication

Is it these issues that caused REST to retreat to the realm of APIs?

Now ajax can help solve these problems so its reasonable to think about making your web app UI RESTful. XHR does support PUT and DELETE (as well as GET and POST). There are a few options for maintaining state on the client:

  • A single page app can keep state in JavaScript objects
  • Gears or HTML 5 client storage can store session state and more
  • HTML 5 session history can handle specific issues with the back button

Authentication is still an issue and the solution to CSRF requires server side session state. I need to think more about these.

Lets assume all the issues with your web app UI being fully RESTful are solved. Now you want to also provide your application’s functionality/data to other arbitrary clients. Do you do this as a completely independent effort? It makes sense to me that the UI should be using the same RESTful data layer API that other clients would use.

Now you have two related but separate planes of URLs; UI URLs and data URLs. You will find yourself in the same situation by following the principals of SOFEA if you try to make the UI and data layer RESTful.

While looking at a list or table of accounts the UI URL might be something like myapp.com/ui#accounts (the hash is used because I’m assuming a single page app) and the URL for the resource supplying the account collection might be myapp.com/api/accounts. The names of accounts in the list include links to details about each account. Clearly, for bookmarking, the UI URL needs to include the account identifier. The UI URL for an account detail view might be myapp.com/ui#accountdetails;083249. The data URL could be myapp.com/api/accounts/083249.

If your RESTful data layer follows the principal of “hypermedia as the engine of application state” then the /api/accounts representation should have (in some form or another) the URLs to each account in the collection. The accounts list UI also needs to know the the UI URL for account details.

To click a link and get from the accounts list to an account detail vew there are two things that must be done. First the accounts list UI must know how to take the account id number and add it to the account details UI URL. There is nothing in the uniform REST interface that tells the UI how to do this. In fact programs should not pick apart URLs (there seems to be some debate about if it is OK for programs to assemble URLs) .

Second the account details UI needs to know the data URL to use for the specific account. Here are some ways it could happen:

  1. The details page could know how to create data URLs taking the account number from its own URL and sticking it in the right place. I think the article “Describing RESTful Applications” is saying this isn’t OK.
  2. The accounts list page could put the data URL somewhere in memory (or other client storage if it isn’t a single page app) so that the account details page can retrieve and use it. But this wouldn’t work for bookmarking.
  3. The UI URL could have the full data layer URL encoded in it like so: myapp.com/ui#accountdetails;/api/accounts/083249. This solves the bookmarking problem but something just doesn’t feel right about this URL. If nothing else its long and repetitive.

Articles on RESTful APIs (including the above linked “Describing RESTful Applications”), put a lot of emphasis on loose coupling between servers — the things providing the API — and clients. There seems to be an underlying assumption that generic or arbitrary clients exist.

In my past experience of building desktop applications the UI has detailed knowledge of the underlying data. In traditional MVC architecture (if there is such a thing) the views know about the model (the views can call methods of the model to get or set data) but the model has no direct access to the views. Some sort of publish subscribe event notification is used by the model to let the views know about changes. The main motivation is to allow multiple different views for the same data. What does this have to do with web apps? Most web apps don’t need change notifications in the form of server push. The point here is that the UI is tightly coupled to the model but the model has no direct dependency on the UI. The model is there to serve the needs of the UI but is not impacted by its implementation. The UI can change independent of the model. If the UI needs some particular new data then it will be added to a resource or if appropriate a new type of resource will be created. If a resource changes, it’s URL renamed or is removed it will likely break existing UI. To me this is the nature of applications. This is why the first option above (the details page knowing how to create the data URL) seems OK to me.

It may be better if the data resource described how to create the URL. It could provide the full URL for the benefit of the assumed generic clients and in addition provide a pattern plus the identifying parts. For example: rel = details, url = /api/accounts/083249, pattern = /api/accounts/*, id = 083249. The trouble is that the description is most efficiently given in the data resource providing the link (/api/accounts) but it is needed by the UI account details page.

It is pointless to have an API, RESTful or otherwise, without some client but is the arbitrary client a myth? I think it depends on the generality of the API. Clearly the Amazon S3 service is very general and can be used by many different kinds of clients. If the data resources come about as part of building a specific web application then they are probably very specific to that UI. While other clients are possible they would probably end up being similar to the original web app or require changes to the data resources. This is a case of different UIs having different needs of its data model.

Is there a general client — one that can consume in a useful way my REST API as well as yours? For this to work the client and services would have to have a shared understanding of some content type. For the client to provide a UI the content type would need to declare the UI. There is such a client and content type. Its the web browser and HTML. Now we have come full circle and can see the problems. HTML has not kept up with the demands of rich efficient user interactions and does not allow separation of the view and model. HTML 5 and XForms may be addressing these issues. (I have only read parts of the HTML 5 Draft and its been a long time since I read XForms.) Should HTML be providing a rich set of built in functionality or a minimal set that can be composed with scripting to accomplish the desired rich interaction? Going back to my HTML is the assembly language of the web analogy, do we want a CISC or RISC architecture?

How are you creating RESTful web apps today?

Writtten language already the best tool for software

Tools, languages or frameworks — which will have the biggest impact on developer productivity? I previously said that I thought languages could do better than frameworks in this regard. Now I’ll talk about at least one class of tools.

A good deal of focus around developer productivity is on visual editing tools. I think these efforts are misguided and unlikely to produce major benefits to developers. Clearly tools are useful. Humans are natural tool makers and users. Written language is arguably the most important tool invented by humans. Programming languages are tools.

So what do I mean by visual editing tools? My definition may be a little fuzzy but here it is: I’m talking about tools for creating software that don’t have an underlying text based language or completely hide it. They usually involve working with pictures, diagrams, or forms. Some examples:

  • Business Process Workflow editors
  • Property editors
  • Visual UI layout editors

These are some of the general problems I have found with visual editors. Not all of them have all these problems and I’m not signaling out a specific tool.

  • Scale — Sometimes the visual editors don’t take scale into consideration. They work well for small programs but break down when used on larger programs. The breakdown may be due to implementation details or a more fundamental limit in ability to comprehend the visualization at large scales.
  • lock in — This is when a tool requires you to use a particular platform, IDE, tool set, library or framework. The same lock in is possible with languages but it tends not to be; at least not for the more popular languages. Perhaps that is what makes them popular. As a developer I’m very skittish when it comes to lock in. It is much easier to share (and reach agreement on) a language specification than it is a GUI specification.
  • Poor source control integration — Being able to track changes to software source over time is very important. Some tools make arbitrary text changes that confound source control systems. The worst example I have seen is a visual work flow editor that changed all ids in its XML format source file every time you saved it. Binary formats are a definite no-no.
  • Lack of flexibility — When writing software in a text based language you are free to write in whatever order your thoughts come to you. Tools that force you to translate your thoughts into specification in a fixed way will end up frustrating users.
  • Inefficient — I tend to find that visual editors get in the way after a while (if not right away). They tend to require a lot of mousing around. Once I know what I’m doing typing is much faster.

Good editing tools assist you while working with a text based language. The most basic editing such as random access to the text, insert, delete, cut, copy, paste etc. is sufficient to get the job done and has the benefit of working equally well for all text languages. Advanced features such as syntax coloring, unobtrusive flagging of errors and warnings, context sensitive completion, access to language and API help, type based searching and navigation, refactoring, template insertion/completion, bracket pair matching, indenting etc. are very useful and don’t replace or get in the way of the text.

Diagrams can help visualize aspects of programs but programs are too complex and multidimensional to be fully represented in any picture or diagram so it is a mistake to think that the visualization is the program and can be edited directly. Being able to draw (or even edit) diagrams is a nice add on for a language but the language must come first and stand on its own.

Some tools claim to have an editable underlying text format when in fact it is just the tool’s internal model serialized as XML. This doesn’t count. Not because its XML but because it tends to favor ease of serialization and the needs of the tool over the ease with which humans can read and write it. XSLT is a good example of a language that uses XML syntax. It was clearly designed to be edited directly by humans. The fact that it can’t be fully described with XML schema is probably a good indication of this. I like XSLT but I also agree with Terence Parr when he says “I implore everyone to please stop using XML as a human interface!” (see his excellent book “The Definitive ANTLR Reference”) .

The best development productivity tools help the user write in a given language not replace language.

HTML is the assembly language of the web

My first computer was a Texas Instruments TI-99/4A. I got it just before going to college. I had been interested in computers since junior high but this was the first time I could afford one. The primary programming options were Extended BASIC and assembly. Writing interesting programs required assembly (because BASIC was too slow) so I learned TI TMS9900 assembly language. I went on to learn a few others including PDP-11 and 8080.

If you have programmed in any assembly language you know how tedious it is. There are so many details to be concerned with that have nothing to do with the problem you are trying to solve — simple instructions, addressing modes, managing registers, and maintaining the call stack to name a few. Because the instructions are so simple you write volumes to accomplish little. Often you would be dealing directly with hardware devices so at the same time as dealing with the many details of the processor’s instruction set you had to understand the idiosyncrasies of the device IO registers.

Once you got some small useful thing working like writing a character to the screen you made it into a reusable library routine so that you didn’t have to write that code again. Libraries helped but the big leap in productivity came with higher level languages. Frederic Brooks wrote in No Silver Bullet: Essence and Accidents of Software Engineering:

“Surely the most powerful stroke for software productivity, reliability, and simplicity has been the progressive use of high-level languages for programming.”

Structured statements replaced compares and jumps. Expressions took the place of numerous instructions. The vast (well not so vast by todays standards) stretches of memory locations were replaced by data structures. The compiler took care of pushing values onto the call stack and popping them on return.

The details were interesting to a point and there was satisfaction in having the level of knowledge needed to program in assembly language. Making the switch to a language like C meant giving up some control but it was well worth it because of the productivity gain.

The switch from assembly to high-level languages didn’t happen instantaneously. Trust had to be won. Early compilers had bugs. People thought they could do better optimization than the compiler, and at first they could, but compilers got better and better. Compilers could also do something that assembly programs couldn’t — they could compile your program so it could run on different types of CPUs.

Writing web applications today reminds me of assembly language programming because there are so many details to be concerned with: HTML, CSS, JavaScript, HTTP, and browser incompatibilities and idiosyncrasies. Then there is the server side technology; pick one of Java servlets, CGI, PHP, .NET etc. and perhaps a framework or two. Also you probably need to know SQL if your application data is stored in a database. Cross cutting issues such as usability, accessibility, security, performance and internationalization add additional dimensions.

The details can be interesting and sometimes frustrating. Do I really need to know that IE make the href attribute an absolute URL when inserting an anchor into the DOM while Firefox does not? (This usually doesn’t matter since if you use the href property you always get the absolute URL but it bit me once.)

I would like to create web applications without having to worry about many of these details. When I say HTML is the assembly language of the web I’m including CSS and JavaScript. I see the Browser as the processor (CPU) and HTML, CSS, and JavaScript as the instruction set. I get the feeling that we should be programming at some higher level.

There are plenty of libraries and frameworks available that help. Some are getting to the point where you don’t need to know much about HTML or JavaScript (GWT, JSF come to mind). JavaScript libraries like jQuery or Dojo are making it much easier to write cross browser code.

So whats wrong? Like I said, some of these frameworks abstract away HTML so you can think in terms of higher level “components”. The problem is they are still just libraries (for the purposes of this article frameworks are just really big libraries). I believe that larger productivity gains are possible from languages rather than libraries. The languages may be domain specific — for creating web applications — but thats OK. The languages could be compiled or interpreted or some combination of both.

One very important thing that happened with the move from assembly to high-level languages is that one relatively small group of people were able to focus on implementing compilers that did great optimization following the best practices for the particular CPU type while another much larger group of people focused on building their applications.

Imagine if best practices like using a hidden token to protect against CSRF or using the POST-Redirect-GET pattern were built into the language.

Imagine if cross cutting concerns and decisions that usually have to be made up front were compiler switches or runtime options. Examples:

  • What level of functionality do old browsers get?
  • Is the back end going to be PHP or J2EE?
  • Do my users have to have JavaScript enabled?
  • Should session state be kept on the client or server?

There are huge opportunities for performance optimizations – on the client, over the wire, and on the server. The optimizations can improve over time independent of the applications. Imagine if css, js, and images were automatically combined, minified, and compressed such that overall response time was minimized.

Yes the abstractions will leak. It will take time for the language implementations to get good enough and for programmers to trust them. I would like to see this happen.

Some may wonder if I actually read “No Silver Bullet” or just pulled a quote from it. After all it says that languages are not a silver bullet. I’m not arguing that they are. I’m saying that they can do better than frameworks. There is enough accidental complexity in building web apps today that a domain specific language could be a big help.

So why hasn’t it happened yet? Perhaps its just that the right framework hasn’t come along yet. Thats the thinking that has given us at least 57 frameworks in Java alone. I think some believe that tools are the answer. It used to be that you could charge money for a compiler. Not so anymore. But that can’t be the problem since most of the frameworks are also free.

Perhaps it is already happening. I have not dug into Links but it sounds just like what I’m talking about.

I plan to write more on this topic in the future.