Interactive Grid: Under the Hood

Last updated on Thursday, January 12, 2017

[This article is about a beta release of APEX 5.1 known as Early Adopter 1 so the details are subject to change]

[UPDATE Jan-2017 APEX 5.1 was released. Don’t use the Early Adopter link. You can now try it out at apex.oracle.com. I made a few updates to this article as marked]

One of the major and eagerly awaited features of APEX 5.1 is Interactive Grid; an editable data table supporting master-detail and much more. You can learn about Interactive Grid by signing up for a workspace at the Early Adopter site, installing the Sample Interactive Grids application, running it and trying out each example. Be sure to read the overview section of each page. I expect in the coming weeks there will be tutorials and other information from a number of people about the features of Interactive Grid and how to use it in various ways. But here I want to talk about its internal architecture.

Why is it important to know about the internal architecture of Interactive Grid? Its not. To use a car analogy most people don’t care to know how their car works. What matters is that it gets them from point A to point B and possibly looks good as well. But some people like to know a little about how the car works, what’s under the hood, even if they never intend to service or build a car themselves. This article is for those who like to know how things work, and I expect there are a fair number of them in the APEX development community.

Interactive Reports have been the workhorse of APEX applications for a long time now and for good reason. They pack a ton of functionality into a component that is easy to configure and use. We took all the best features of Interactive Reports and added editing (plus some other goodies). But doing this required a completely different architecture. Interactive Report generates all the HTML markup on the server as part of the APEX page. The data returned from the SQL query is rendered along with all the other markup that makes up the UI; the toolbar, the dialogs, everything. The client side behaviors are implemented in a single monolithic JavaScript module. In contrast Interactive Grid does virtually all the HTML rendering on the client. The server provides the data in JSON format. Edits are saved to the server in JSON format as well. The client side of Interactive Grid is actually a collection of several coordinated widgets and other facilities that are the focus of this article.

The following diagram shows all the major parts that make up the client side of Interactive Grid.

The rest of this article explains the diagram and then goes into how Interactive Grid has affected a number of other aspects of APEX.

Each blue rectangle represents a widget or more specifically a UI widget based on the jQuery UI widget factory. The dotted border around ojChart indicates that it is not built by the APEX team but rather comes from the Oracle JavaScript Extension Toolkit (JET) library. The yellow rectangles represent other modules. The green rounded rectangles are instances of the type managed by the module they are in. The wide arrows represent data flows between the client and server using ajax. The black closed arrows indicate that one component owns, calls, or uses another. The red dotted arrows represent notifications. The blue open arrow indicates that the grid, recordView, and tableModelView widgets are all derived or extend from tableModelViewBase. The blue double arrow represents data flow between the models and UI edit controls.

To keep the diagram simple a number of modules are not shown including stickyWidget, collapsible widget, apex.lang, apex.util, a new one apex.message and jQuery UI dialog widget. With the exception of menu, iconList, apex.item, and apex.actions all the modules shown are brand new and created to support Interactive Grid. A note about names: I’m using names for JavaScript widgets and APIs as you would refer to them in code. It should be straight forward to find the corresponding JavaScript file for those that want to dig deeper. All the files are in the sub folder libraries/apex. A namespaced APIs such as apex.util is in util.js. A widget such as recordView is in widget.recordView.js. One exception is that tableModelView is combined with tableModelViewBase in widget.tableModelViewBase.js.

First of all Interactive Grid is a region plug-in. A native one, meaning that the implementation is part of APEX, but still a plug-in. For saving changes there is a companion native process plug-in called Interactive Grid – Automatic Row Processing (DML). During page rendering the region plug-in adds a single div for the interactiveGrid widget and markup for each of the column items that will edit cell values (more on this later). Then at the end of the page it adds some JavaScript initialization code that creates the interactiveGrid widget with all of its configuration and depending on the Lazy Loading attribute may include some initial data as a literal JavaScript object structure. The configuration comes from Interactive Grid attributes and column attributes that are edited with Page Designer and saved report settings that are persisted on the server as the user or developer customizes the Interactive Grid at run-time.

InteractiveGrid Widget

The main function of the interactiveGrid widget is to create and control a number of other widgets, disseminating configuration settings among them. It also manages the saved report settings retrieving them from and saving them to the server. It has very little direct responsibility for the UI. It gets other widgets to do the UI rendering and user interaction on its behalf. One exception is the report settings area just under the toolbar, another is the column popup menu. There are also a number of dialogs for configuring report settings such as filters, aggregates, and computed columns [UPDATE Jan-2017 computed columns did not make it into 5.1]. The interactiveGrid widget defines the content and behavior for these dialogs but the common implementation is handled by jQuery UI dialog, reportView, and grid widgets using apex.model.

Toolbar Widget, Menu Widget and Actions

At the top of the Interactive Grid is a toolbar that looks somewhat like the one in Interactive Report but the implementation is completely new. One thing people have wanted to do with Interactive Reports is customize the toolbar and/or Actions menu and in some cases, by making unsafe assumptions about the markup and internal details of Interactive Report, they accomplished this. (In APEX 5.0 there are reasonably safe ways to customize the Actions menu using the menu widget API. hints) The new toolbar widget makes it possible to completely configure or even replace the Interactive Grid toolbar. For example, you could add controls at any position or remove existing controls and even change the behavior of existing controls.

A toolbar consists of an array of groups of controls such as buttons, menu buttons (and the corresponding menu), toggle buttons, radio groups, etc. Menu buttons use the existing APEX menu widget and menu button functionality. The menu widget option structure is incorporated into the toolbar option structure. The behavior and state for each toolbar control is handled by the apex.actions facility. Both toolbar and menu are integrated with actions. The toolbar widget is also used by reportView (single row view) and in some Interactive Grid dialogs.

The APEX actions API was introduced in APEX 5.0. It manages state and supports keyboard shortcuts for actions. I described it in this post about adding keyboard shortcuts to menus. For 5.1 apex.actions was enhanced to support the needs of toolbar widget and Interactive Grid. Most importantly the ability to scope actions to a particular region was added and also support for radio group actions.

Interactive Grid View Interface

Below the toolbar and report settings area (only shown when there are report setting customizations) is the main content of the Interactive Grid. This can show a number of different views: Grid, Icon, Detail, Chart, and Group By [UPDATE Jan-2017 group by did not make it into 5.1]. The interactiveGrid widget deals with each of these views in a generic way by defining a view interface and implementing a view object for each kind of view. Each view knows how to control its corresponding widget. This greatly simplifies interactiveGrid because it doesn’t have to know details about how each view widget works or is configured it only deals with the view interface and treats each view interchangeably. Some views use the same underlying widget. The Grid view and Group By view both use the grid widget. The Icon view and Detail view both use tableModelView widget. In both cases it is differences in configuration that define the view. (In the diagram the Group By and Chart views are in italics because they are not yet implemented in the beta.) The Single Row View in spite of its name is not really a view like the others. It is an alternate presentation of the Grid view.

Models and Views

Views are responsible for presenting data and letting the user interact with that data but they don’t actually store or manage the data. That is the job of the model (in our case apex.model). This is known as model view separation and it is an important architecture used in desktop apps and even modern web apps. Views have a direct reference to their model. They get data from the model and if they change the data they tell the model about it. Models on the other hand have no idea about the views that are using it. It could be used by zero or more views. Models notify views of changes using the observer pattern. Any number of views (or other interested parties) can register with a model to receive notifications when the model data changes. This means that the same data can be presented in different ways. For example the Grid View (grid widget) and Single Row View (recordView) are two different views of the same model.

Some notes on terminology:

• The term view is used in a number of slightly different ways. Generally when talking about models and views the view is anything that presents the data contained in the model to the user. The grid and recordView widgets are concrete examples of views. The view interface and more to the point the concrete instance objects in the interactiveGrid are wrapper adapters around the concrete view widgets, so in the context of model-view are not actually views. In the UI there is a view menu that lets you switch between views such as Grid and Chart.
• People familiar with the term MVC (Model-View-Controller) may wonder if that is what model view separation is getting at. Basically yes. I have avoided the term MVC because there is a great deal of variation in what it means from one person to the next. You can think of the view widgets as being both view and controller combined. We also informally refer to widgets such as interactiveGrid or page designer modules such as f4000_p4500.js as as controllers because they coordinate the actions a number of other widgets.
• People with a database background may be confused by this use of the term model. They may think of a model as an abstract description or representation of a system’s structured data or schema. In this context a model is a concrete object that holds data. This model stores data on the client temporarily for the benefit of the UI or presentation layer. The persistence layer is typically the Oracle Database.

apex.model

The APEX model is a generic data store that is configured with the details about the specific data it is going to store. For Interactive Grid it stores table shaped data (a collection of records with each record made up of fields, also referred to as rows and columns). The model also supports a single record or a tree as the model shape but these are not used by Interactive Grid. The record itself can either be an object where the fields are the object properties or an array where each field is at a particular array index. Interactive Grid always uses an array because the JSON representation uses fewer bytes and is therefore more efficient to send and receive. A record can also include metadata such as highlight information, allowed operations, and fields that are read only. The model itself keeps metadata such as record or field error flags and if the record was edited, inserted or deleted.

The model supports a number of operations such as delete, set field value, insert, copy, move, revert, refresh, fetch more records, etc. The API is fairly stable; see the source file doc comments for details. It supports access control to say what can be done to a record, for example allow edit or allow delete. This can be determined by the record itself (see Allowed Row Operations Column attribute) or using a type system where a field/column is designated as supplying the type and the model is configured with a structure that maps types to the allowed operations. Interactive Grid does not use types by default. The type system was taken from the APEX treeView and intended for its use in the future.

The model is responsible for fetching data from the server and saving changes back to the server. When a view asks the model for data that it doesn’t yet have the model will fetch it from the server. The view and interactiveGrid widgets are not involved with the details. The model knows if it has any changes and also if there are errors. You can call methods on the model to access this information. When a model is saved, either by the Interactive Grid or when the whole page is saved, just the changed records are gathered up and converted to JSON to send to the server. The record data sent includes metadata that tells if the record was edited, or inserted or deleted. Record metadata also includes checksums so that only the data that is allowed to be changed can actually be changed.

The model supports view features such as pagination, control breaks, and aggregation records. It also has methods to subscribe (and unsubscribe) to receive notifications about changes to the data.

Models support master-detail scenarios. A model knows if it has a master model and what record of the master the model is associated with. There is a single detail model for each master record. This is coordinated by the interactiveGrid widget. When a record is selected in the master a new detail model instance is created if needed. A cache of model instances is kept with the least recently used model instances kicked out to make room for new ones. The default cache size is 10. However if a detail model has changes then it will not be removed. You can see the cache in action by observing the progress spinner. Click one record in the master grid and notice the spinner as data is fetched. Now click another master record, again there is a progress spinner. Now click back on the previously selected record. Notice that there is no spinner because the existing model from the cache is used. (Unfortunately in the beta the cache limit is broken so if you click on 11 different master records then click on the one selected first you won’t see the spinner like you should.) When the master grid is saved or the whole page is saved all the models associated with the master that have changes are also saved.

The interactiveGrid widget manages the views (through the view interface) and also the models for each view. Each view has its own associated model as can be seen in the above diagram.

Grid Widget

The grid widget is a data table that supports fixed headers, frozen columns, scroll paging as well as traditional paging, and editing. These are not built-in features of HTML tables. For example, when there are fixed headers and frozen columns there are actually four separate but coordinated HTML tables. Scroll paging means that what you see is just a window or view-port into all the data. As you scroll new rows are rendered from the model as needed. If the model doesn’t currently have the data it requests more data from the server (without necessarily throwing out the old data). There are a number of other features such as drag and drop column reordering but these are the critical features that dictate the architecture of the grid widget.

The old APEX Tabular Forms rendered an edit control for each and every cell. This is a usability and accessibility problem. If you just want to navigate the table you have to tab through every cell. The solution is having distinct navigation and editing modes. In addition when there are a very large number of rows and/or columns it is a performance problem to render edit controls for each and every cell (because the size of the DOM is drastically increased). For this reason the Grid Widget uses the flyweight pattern. A single hidden edit control is rendered for each column. Then as the user focuses a cell in edit mode the corresponding edit control is swapped into the cell and the control’s value is set to the value of the cell from the model. So no matter how many rows are rendered there is always only one row of edit controls. How a cell is rendered when not being edited is quite different from how it is rendered for editing. But in both cases the underlying value comes from the model.

By now it should be clear how having a separate model layer is necessary for scroll paging and editing.

Another problem with Tabular Forms is that they did not use APEX page items as the edit controls. There were, for example, separate implementations for text fields for Tabular Forms and text field page items. For Interactive Grid we have extended item plug-ins to support both page items and column items. The same item can be both a page item and a column item (you have to opt in to this new capability). We have tried to minimize the distinction between the two. The apex.item API has been extended to meet the needs of column items. This provides consistency and extensibility. It means that it is possible to create item plug-ins that can be used in Interactive Grid. This is also important for supporting Dynamic Actions and Cascading LOVs since these are features of page/column items and not the apex.model.

RecordView Widget

The recordView widget presents a single record (row) of the model for viewing or editing in the style of a form. It allows you to go to the next or previous record. It is designed to share the same model with a grid widget or in theory any other widget that uses a model. It uses the same column items to edit the record fields (cells). The difference between grid and recordView editing is that with the latter all of the column items are visible at the same time. Because the grid and recordView use the same column items only one can be editing at a time. This is not a problem in Interactive Grid because only one of the two is visible at a time.

Much of the code that deals with interacting with the model is in the base widget tableModelViewBase, which is shared with grid widget. It is responsible for rendering the display (read-only) view of a cell, moving data from the model to the column item for editing, and moving data from the column item back to the model when done editing.

TableModelView Widget

The tableModelView widget is a very simple layer built on tableModelViewBase. It uses client side templates to specify the markup to render: before the rows, for each row, and after the rows. The data used by the row template comes from the model. It is used to implement both Icon view and Detail view. Detail view has no additional behavior beyond pagination. Icon view uses the iconList widget to implement a 2D list that manages focus so you can use arrow keys to navigate the list and also supports single or multiple selection. The tableModelView adds support for pagination that iconList doesn’t have on its own.

As has already been said the base widget deals with model interactions. This includes all types of pagination. This is how grid, icon, and detail views all support the same pagination options.

Summary

The above is a brief introduction and overview of each of the client side modules that make up Interactive Grid. The benefits of this modular design are that each component can be separately developed, maintained, and tested. In fact we use static HTML pages (outside of APEX) to develop and test each widget and unit tests for most of the other modules. It also allows us, going forward, to reuse the different pieces in other contexts just as the iconList and menu widgets, new in APEX 5.0, have been reused in a number of ways and in a number of places. No details or promises but given what you now know it should not be a surprise if, for example, you see the toolbar widget show up in other places.

As was previously mentioned we have already gotten extra use out of apex.model and various widget views in the interactiveGrid dialogs. Many of the new Interactive Grid dialogs have been redesigned to allow you edit multiple report setting instances at once. For example you can edit several filters at once including adding and removing filters. Then if you are happy with your changes you click Save and if not you click Cancel. This is a perfect use for a model. Changes to temporary data in the dialog can’t affect the interactiveGrid report settings until the user chooses to save. When the filter dialog opens it loads data from the internal report settings into a model. Each row/record represents the settings of a filter. Then the user views/edits any or all of the filters. When the changes are saved the filter dialog stores the changes back to the internal report settings. All the common functionality of these dialogs, including creating the model and creating and configuring the dialog, grid, and recordView widgets, is handled by $.apex.recordView.createModelEditDialog.

Interactive Grid is highly customizable and controllable using a JavaScript API. There are already a number of examples in the Sample Interactive Grid app. In this Early Adopter release the API is not yet stable. In the 5.1 release the API is not planned to be documented or supported. Not supported just means that it could change in a future release that would require you to make changes to your app.

It should not be a surprise that the architecture will affect how you use Interactive Grid programmatically. For some things such as adding filters or getting the currently selected records of the current view you will call a method directly on the interactiveGrid widget. For actions that are available on the toolbar or menus you will first get a reference to the apex.actions interface and then invoke the action by name. In less common cases you may need to interact directly with one of the view widgets or a specific model. In this case you will first ask the interactiveGrid widget for the specific view widget or model and then call methods on the view or model.

Wider impact

Finally I want to tell you about a few ways in which Interactive Grid has affected other aspects of APEX.

Interactive Grid allows you to use templates to define the contents of a cell. For example using a column Type of HTML Expression. Another case is with the Link Text attribute. The Icon view and Detail view also use templates but for the whole record/row. Because the client model can be edited and because rendering in general is done on the client these templates must be evaluated on the client. This is very new and different for APEX, which has always evaluated templates on the server. The template syntax uses the familiar &<item-name>. syntax used by server side APEX templates. For details, see the doc comments for apex.util.applyTemplate. This same template function is used internally by the new apex.message API. You may come up with your own uses for applyTemplate independent of Interactive Grid.

Tabular forms used the Fxx arrays to save data. Interactive Grid could have gone the same way but it would have been too limiting. For example we needed to have multiple grids on one page. Instead we completely reworked how APEX pages are submitted. Form and model data is serialized as JSON, chunked, and sent as the p_json parameter in a normal x-www-form-urlencoded (or multipart/form-data) POST request. This affected the apex.page and apex.server modules as well as the server wwv_flow.accept procedure and new wwv_flow.ajax procedure. Note the new apex.server API is not yet stable. [UPDATE Jan-2017 see the APEX JavaScript API documentation for details on apex.server namespace] This has had many side benefits such as removing the 200 page items per page limit.

The very nature of grid editing means that a lot of changes can accumulate on the client. This leads to two problems. 1) having to wait until you are done with all your changes for the server to validate your changes. 2) forgetting to save your changes by accidentally navigating away from the grid page. Interactive grid was a strong motivating factor in supporting client side validation and warning on unsaved changes. Because Interactive Grid regions can be mixed with ordinary form regions on the same page the user will expect the same functionality to apply to every editable thing on the page so we had to come up with general solutions.

Client validation is based on the HTML 5 validation API. This means that declarative HTML validation attributes such as required and pattern will work with applicable APEX items (doesn’t matter if it is a page item or a column item). The validation functionality is exposed through the apex.item API with the getValidity and getValidationMessage methods. This means that your own item plug-ins can also support validation. For details see Sample Interactive Grid app Form with Grid, Validation, and Client Validation pages. For form validation to happen on the client when the submit button is pressed the Execute Validations attribute on the submit button must be set to Yes and Application Compatibility Mode must be >= 5.1.

To reliably warn the user about unsaved changes in a web browser the beforeunload event must be used. This is the only why that catches all cases of leaving a page. But there are a number of details involved in making this work well. First you need to know if editable data on the page has changed. I already mentioned that models know if they have changed so the beforeunload event handler just needs to ask them. For form controls the apex.item API is extended to include the isChanged method. This will allow customer or third party item plug-ins to participate in this feature. The next thing is to be able to configure specific submit or navigation buttons that you don’t want to warn. For example a Submit button that is going to save the page shouldn’t warn about unsaved changes because that is exactly what the user is doing. Neither should a Cancel button because the user is saying that they don’t care about the changes. There are some items that should not cause a warning. For example a select list is only used to hide/show or enable/disable different sets of edit fields. This is why there are Warn on Unsaved Changes attributes at the page, button, and item levels.

I hope this article has given you a greater appreciation for the Interactive Grid and perhaps given you some ideas about advanced ways you can employ it in your apps.