Introduction

Abstract

Background description

OpenMRS web application includes an initial setup and update wizards those greatly simplifies DB creation, configuration and update on web-app startup.

But, when user is running installation or the database update wizards the GUI messages, on the pages which are rendered, aren't translated into the supported languages except english.

Goal

The main aim of this project is an integration of an internalization tool for velocity templates and localizing the initialization and update wizards.

Localization style must be alike desktop's application with choosing preferred language at 1st step of setup wizard, with ability to move back and change it while wizard is running and with storing that selected language by app for further using it.

Goals for midterm

The selected tool should have been integrated and have a few pages localized
The user should be able to select the language at startup and persisted to the database

Final goals

The user should be able to move back and forth through the wizard's pages even when the text displayed is in a language they don't understand so well or if they selected a wrong language by mistake
Have all pages localized
Have qualitative documentation

Understanding the basic principals of initial setup and update wizard design

Above mentioned wizards are running when OpenMRS web-application is starting.

The initial setup wizard will run either when web-app starts after first deploy or when application data directory has changed. In turn, update wizard will be runned only if there are some new changes within liquibase file which are need to be applyed to database.

For achieve of such behaviour, OpenMRS uses appropriate servlet's filters. Currently those are the InitializationFilter and the UpdateFilter. After the app's servlet's context has initialized, context initialization listener will call method setupNeeded(). And if this method returns true, it means that the filters needs to take some action and OpenMRS won't start yet.

And since initial setup (or database update) is needed and each request to openmrs web-application are being filtered by InitializationFilter and/or the UpdateFilter filters, then the first GET-request will cause start execution of those wizards.

Appropriate doGet() method will be invoked for this GET-request . That method renders first page of any wizard into the server output stream. This page is rendering from velocity template. Each wizard consists of set of sequenced pages. As mentioned above, every such page is rendering from appropriate velocity template.

When user is moving through wizard's pages he, preferably, sends POST-requests to server, which are processed by doPost() methods of filters. Depending on previous step of wizard, this method will render next corresponding page via appropriate velocity template. And all GUI messages, on the pages which are rendered from templates, aren't translated into the supported languages.

Schedule

Proposed time-line:

This is detailed time-line that contains plans of project's execution and it's score.

10 April – 23 April

More thoroughly familiarize myself with the code and the community, the version control system, the documentation and do test of system;
Familiarize myself completely with installation and update wizard's functionality and architecture.

26 April – 22 May (before the official coding time)

To make a review of existing localization tools for velocity templates with providing relevant analyzes document for mentor
To discuss this document and explore all bottlenecks within project to become absolutely clear about my future goals and the way of it implementation
To determine final project's architecture and design of appropriate classes (provide necessary UML-diagrams, those describe structural and behavior aspects of project)
To discuss those designed diagrams with mentor.
To do self coding with installation and update wizard projects to improve my further understanding and ease of use;

23 May -- 18 June (official coding period starts)

Write code of appropriate classes considering with designed diagrams.
Define entire set of messages to be localized for installation and update wizards.
Perform a full translation for one of the the supported languages (eg Italian), to be able to right away test the functionality that is being developed

19 June -- 1 July

Make the necessary changes with wizard's pages (notably, add page for selecting the preferred language and add appropriate control buttons and also test new navigation between wizard's pages)
Provide new feature for storing/retrieving user selected language.
Write appropriate changes into the install/update filter classes to ensure direct integration of existing velocity templates with developed on previous steps custom localization tools.

1 July – 19 July

Test received functionality for two languages (English by default, and previously the chosen one) and bug fixes.
Perform further translation of interface messages into other supported languages.
Prepare project for midterm evaluation.

JULY 15th - MID TERM EVALUATION

20 July – 25 July:

To be in constant touch with the mentor and to let him know about our progress.
Making further changes in the code to improve the functionality, exception handling, bug removal. Most of the time will be consumed for rigorous testing and bug fixes.

26 July – 31 July:

For writing documentation

1 August – 15 August:

A buffer of two weeks has been kept for any unpredictable delay.

Project's requirements

So, above mentioned goal can be successfully achieved only if developed project will satisfy next requirements:

Functional requirements

Appropriate page should be shown for user at first step of the setup wizard;
That page should contain selection list with all possible languages, checkbox for indicate that the selected language should be persisted after wizard’s finish and navigation element (next arrow) for continue;
Selection list should be dynamically filled with languages basing on existing web app's message resource files;
The user should be able to move back and forth through the each wizard's pages by using appropriate navigation elements (arrows are preferred in case when the text displayed is in a language they don't understand so well or if they selected a wrong language by mistake);
When user is moving through the initial setup wizard's pages, language should not be changed until user will return to 1^st page and change it;
Selected by user language should be saved into cookie (or as http session's attribute) during wizard’s work and (if user have marked checkbox), it should be persisted into DB as admin’s property;
When user is running the database update wizard, all pages should be displayed in previously persisted language or in English by default.

All functional requirements to the system can be formalized by using following use case diagram:

Non-functional requirements

Support for new languages should not require recompilation;
Access translated text with resource bundles;
Resource bundles should be loaded as fast as possible (on web app startup or at first request).
Textual elements, such as status messages and the GUI component labels of both wizards should be completely translated into most of supported languages (Italian, Spanish, French and Portuguese);
Employ the use of previous arrow and next arrows as navigation elements instead of buttons with text ;
Only Unicode encoding characters should be used for translation;

After describing of requirements we can make more desictive steps. As we need to localize messages through velocity templates next in chapter contains analyze of existing L10n tools.

Analyze of existing L10n tools for velocity templates

Overview of existing L10n tools

Based on the above requirements to ensure L10n support for velocity templates, it's need to use an existing java properties files as message's source. Those properties files will contain translation of appropriate GUI text messages of both wizards. Those properties files are located under web app's /WEB-INF directory.

Considering the above mentioned, one of the project's central tasks is a loading message resources through Java's message resource bundle by employing some form of message localization tool. Such tool should be like a connecting element between localized messages from one hand, and velocity pages from other hand. That is why it's necessarily to pay a special attention for such tool's design.

After searching through the Internet I found several widely-used tools. When I analyzed them, I used next criteria:

ease of use via velocity
based on Resource bundles
wide extensibility and good customization

So, I decided to do complete overview of the next tools which met most of criteria in one way or another:

IncludeTool from Velocity-View extension
Spring's tool for Resource Bundles (MessageSourceResourceBundle)
ResourceTool from Generic-Tools

All these tools have a lot of different features and any of them can be chosen.

1) First, and as I think, not the best way is IncludeTool. This is a class from VelocityView which allows for transparent content negotiation. It’s really ease of use via velocity. Usage it “tool” from a template would be something like the following:

#parse( $include.find('header.vm', 'it') )
#include( $include.find('installmethod.vm', 'it') ).

Notably, that it’s not based on Resource bundles (or based deep under the hood). But anyway, in my opinion, the main drawback of this approach is that we must keep separate template file for each supported language. So, using of this approach will complicate adding new functionality (e.g. adding new supported language will require creating one more copy of all templates). So, as result, I can say that this approach doesn’t provide wide extensibility.

2) Second, and much better way, is to use velocity ResourceTool. This is a tool for accessing Resource bundles and formatting messages therein. It's an access point from Velocity templates to standard mechanisms for java support i18n and L10n with Resource bundles. It loads messages from Resource bundles within application’s classpath. For using it we only need to configure velocity toolbox. After configuring it we will gave an opportunity to access to our messages from bundles in this way:

$locale.continue                      ->  Continue
$locale.install.method                ->  Advanced
$locale.install.method                ->  Simple

So, as we can see it also ease of use via velocity and, moreover, it’s based on Resource bundles.

But it also has one drawback. As mentioned above, this tool loads bundles only from class path. And it’s unacceptable, ‘cause our messages.properties files are located in /WEB-INF directory, which, unfortunately, is not included into the classpath, but /WEB-INF/classes and /WEB-INF/lib included (see Fig. 1). But fortunately, we can solve this problem by writing an appropriate tool that is based on existing Resource Tool (we only need to make a substitution of used by default class loader onto our custom).

3) The third approach is Spring’s MessageSourceResourceBundles that allows for accessing a Spring MessageSource as a ResourceBundle. Used for example to expose a Spring MessageSource to JSTL web views. Unfortunately, it also loads resource bundles from class path. Nevertheless in can be rewritten in the desired manner (like MutableResourceBundleMessageSource from core project), but it's not the best approach in this case to using spring.
Such behavior (loading resource's bundles extremely from classpath) can be explained that both velocity ResourceTool and spring MessageSourceResourceBundles uses the same mechanism of loading resources – implementation of java.lang.ClassLoader which loads classes and resources directly from java class path.

Pilot choice for the OpenMRS project.

I made pilot choice of basic localization tool, it was velocity ResourceTool, ‘cause it meet all requirements and it’s also lightweight and flexible.

Few words about it.

Using of this tool might require writing appropriate resource bundles loader for using it within custom tool, that extends velocity ResourceTool.

Which advantages brings up this choice? We won’t change directory location in OpenMRS war file (suchlike in case of classpath-based resource loaders) or add new template files, or just even create separate spring application context. Nothing that we do not have to do.

What we should do? We just simply need to create two own classes (first, that will be responsible for loading and caching resource files from specified location (either from classpath or file system), and another class (should extends velocity ResourceTool class), which will be responsible for getting messages from these bundles by key).

And usage of it approach from velocity template would be something like the following:

$locale.continue,

where, "locale.continue" – message key.

Also, it approach very flexible, cause we simply can change resource bundles access object for new one, or use new similar approach to access it bundles. Moreover, in this case we will only inject our custom message localization tool into InitializationFilter or UpdateFilter classes with the fewest changes with the code therein.

Project's design

So, after we have made overview of subject domain (i.e. localization tools) we can start project design. It involves design of project components structure and their behaviour.

Design of project structure

Initially, the project structure can be represented as a set of interrelated components. Links between these components may be different. The general project structure is shown in the figure below.

UpdateFilter and InitializationFilter are the components which are representing an appropriate existing servlet filters. Basically, for achieve project’s goal it is necessary to make changes within these filter components. In detail, these changes will be described in next chapter of current document.

Next components are named as messages.properties*. These are text files which will contain localized messages for both wizards. Those files are located under /WEB-INF directory of OpenMRS web app (see Analyze of existing L10n tools for velocity templates). Example of their content is shown on related text notes on diagram above.

Another key component of structure is custom localization tool. This is a central component of the project structure. All magic will be around this component. It accesses resources bundles and obtains messages therein.

Such component will be implemented as sub-class of velocity’s ResourceTool class. Creating of sub-class, instead of using standard tool can be explained with need to load resources outside the web-app classpath. For achieve this we will simply override get(Key) method, that is responsible for loading the resources. When we will implement this method ,we should take care about loading messages.properties* from file system by specified path.

Using of this approach will allow as to type $locale.install.method instead of $l10n.get(‘locale.install.method '), for showing localized content.

Also component will contain map with resources bundles for each locale. Moreover, it should be configurable. This means that this component should provide opportunity to easily change its parameters in run-time.

And the last important component is filter utility. This component should manage locales parameters. It will check whether need to save user’s selected language, will retrieve it locale from http session if need, and will store it into DB as admin’s user property. Also it will try to retrieve that persisted language in case of DB update wizard running.

Since we will use custom velocity resource tool we will need to configure appropriate toolbox. Admittedly, this component will manage configuring of localization tool by using java API. For this we will use EasyFactoryConfiguration class from velocity framework. Filter utility will be implemented as java class (or set of classes under the same package). Access to localization tool component will be provided through this one.

Design component’s behavior

In this section the most specific aspects of project coponents behaviour will be described.
This is general statecharts for both, initial setup and update wizards. This is a scheme of user interaction with system's functionality. This diagram shows how the user performs wizards, moving from page to page. Unlike the site maps and/or navigation charts, this diagram shows detailed process and describes defining moments of behaviour.
Statecharts which are showed on picture above, inter alia, contains states to those are worth to pay attention. For initial setup flow's statechart this is a state that is named Storing selectd language. Detailed state diagram, which is covers the main things of this state, is shown below:

Diagram above shows us which steps its need to perform to persist user selected language. As you can see from that diagram, storing occurs after successful completion of the installation process. Otherwise, if some error has occurred during the installation, language would not persisted.

So, first step is retrieving auxiliarity parameter (lets call it "remember my choice") from http session. If this parameter exists in http session's attributes, it means, that appropriate checkbox has been checked by user at first page of initial setup. Next, it's need to pull locale attribute out of http session. Since intallation has been completed succefully, and the OpenMRS context is already loaded and initialized, we can use it to persist locale parameter as admin's global property.

In turn, for update wizard, it's worth to pay an attention to state, when making the choice of which language will be used. Belown diagram describes flow for this state.

Foremost, it's worth to pay an attention, that this flow is happening before wizard running, and it imposes some restrictions. More simply, it is not possible to use openmrs context in this case. So, possible solution will be creation JDBC connections and using raw SQL statement. First, it's need to try to retrieve global property, which, admittedly, has been persisted at last step of installation wizard. To accomplish it, need to establish jdbc connection, prepare raw sql statement and execute it. If execution returns not null result, this means that selected before locale is retrieved and now it's possible to configure velocity toolbox and our custom resource tool for using resource bundle for this locale. Otherwise, if null was returned, it's need to execute one more sql statement to retrieve default locale param.

Project implementation

Before beginning of process of project design we have had an abstract description of problem but didn't have an answers how to solve that. And now, when the process of project design has been finished and we are having solution of problem. With described above solution, we can start project implementation. It involves implementation of project classes, integration them into existing wizards and performing translations of GUI messages for wizards.

Implementation of project's structure

Here is the class diagram of the project. Logically it can be splitted to two parts:

classes, which are related to custom localization tool component - CustomResourceLoader and LocalizationTool. These classes were developed from scratch;
classes, which are represents servlet's filters component and used as wizards core engine - StartupFilter, InitializationFilter and UpdateFilter (these classes were changed during project implementation), and developed from scratch - FilterUtil class.

Lets make a review of those classes with describing of their noteworthy features.

The first and the most important class in project structure is LocalizationTool. This class is intended for accessing ResourceBundles and formatting messages therein. For such porposes we also could use pure velocity's ResourceTool class, but it's not possible, because messages properties files are located outside web application classpath. In fact, this class extends velocity's ResourceTooland uses CustomResourceLoader for accessing messages from resource bundles, which are located outside the classpath. It overrides only two methods from super-class. First of these methods is get(Object, String, Object):Object is intended for obtaining message translation by specified key. Next method, is insert(Object):Key is used to support place holders for compound messages.

Notably, that there is only one instance of this class for one http client's session. This is because multiple users can run, for example, installation wizard at the same time, and each of them can use his own language.

Next class, which was already mentioned above, is CustomResourceLoader. This class is responsible for direct loading resources from file system. It also works like cache of messages files. When creating instance of this class, is making an attempt to load all possible messages files from specified file system location path. Messages files are loading into map as resource bundles, where locale of message properties file is used as key, and resource bundle as value. Further, this map will be used by multiple instances of LocalizationTool class. It worth to pay an attention, that, since all instances of LocalizationTool class perform read-only access to resource bundles cache, there is only one instance of this class per application. This is done to keep low memory consumption.

The StratupFilter is a general class for all servlet filters which are related to wizards. It's a heart of each setup wizard This class contains base functionality for:

managing velocity engine and rendering templates,
configuring tools within velocity toolbox,
properly gathering errors, which are occurred while wizards running;
describes general abstract methods to be implemented by subclasses.

The highlighted features were added into this class as new functionality. As for configuring tools within velocity toolbox, it done in way of support using of any wizard by multiple users at the same time. During running of wizard each client will use his own toolbox. This is because different users can use different languages when working with wizard, and need to configure one custom localization tool for each client. For this purposes there is a map with client's session ids and velocity toolbox's contexts (with custom localization tools therein). So, when client begins wizard, this class creates new entry within this map, where as key used id of http session, established by current client, and as value used context of velocity toolbox, configured for this client. It worth to pay an attention on next thing, when rendering templates, toolbox context is retrieved from that map and is merged with velocity context. Velocity toolbox context is configuring not directly by this class itself, but by its subclasses. For this, any subclass uses configureVelocityToolContext(String locale):ToolContext. So, when such need arises any subclass should:

get current client's locale and his session's id;
using this parameter, (re)configure toolbox context.

As for feature of gathering errors, it also based on map. Using of map in this case can be explained by need of handling compound error messages. Compound messages consist of static and dynamic parts. In the most cases, they are related to exceptions. So, as keys for this map are used keys for those message's and as value are used dynamic parts of messages. For example:

{ key : "install.error.connVerify", value : "admin, javax.sql.Exception ..." }

where:

install.error.connVerify=User account {0} does not work. {1} See the error log for more details

or (for portuguese variant):

install.error.connVerify=Conta de Usuário {0} não funciona. {1} Veja o log de erro para mais detalhes

And next, when rendering template, the key install.error.connVerify will be processed by custom localization tool, corresponding message translation will be retrieved (for exapmle, portuguese translation), and place holders (0) and (1), will be filled by admin and javax.sql.Exception ... values.

Both, UpdateFilter and InitializationFilter classes extend StartupFilter. All functionality related to managing locale related parameters and obtaining client's session id were added into these classes. Each of those classes has methods, which makes decision what language to use in different situation. As for InitializationFilter, its corresponding method makes such decision before rendering first page of initial setup wizard. As for UpdateFilter, its method makes such decision twice (before first page loading, and after user authenticated). Also, those classes are responsible for persisting/restoring locale related parameters (such as system default locale, user's locale property).

For this purposes they use static methods from FilterUtil class. This class contains convinient methods for storing/retrieving locale parameters into/from DB as admin's user property and as default locale property for OpenMRS system.

Code samples

This section contains the most important code snippets, which are describing different aspects of projects functionality implementation. Each part of followed code is shown to describe related functionality as much as clear, so, it can differs from source code, because some parts of code can be ommited.

Notably, that it was very important to make obtaining of different messages translations from velocity templates as simple as possible. It worth to pay an attention on that fact, that all GUI message within wizards logicaly can be splitted into two groups: static labels messages and dynamic error messages. The ways, how we accessing them within templates are almost the same. Basically, we are using appropriate method of localization tool object that is pushed into template. Next code snippnet shows this:

<td colspan="2"><strong>$l10n.get("install.method")</strong></td>

Example above related to rendering static labels message translation. As for showing of error message translation, the next example demonstrates it:

#foreach ($error in $errors.entrySet())
    #if (!$error.getValue()) 
        <li>$l10n.get($error.getKey())</li>
    #else
    <li>$l10n.get($error.getKey()).insert($error.getValue())</li>
    #end
#end

For showing error messages we are iterating over map of error messages. As key this map uses property name of static part of message translation and as value - list of dynamic placeholders for that message. So, if messsage translation doesn't contain any dynamic parts, it will be rendered alike normal static message (see line 3). But if error message translation contains some dynamic entries, they will be placed into message by calling insert(Object[]) method of custom localization tool (see line 5).

As you can see, through template all messages can be obtained via custom localization tool object (i.e. l10n on examples above). Considering with components behavior for using custom implementation of any velocity tool within velocity templates, we need to configure velocity tool context. Next sample shows that:

// first we are creating manager for tools, factory for configuring tools 
// and empty configuration object for velocity toolbox
ToolManager velocityToolManager = new ToolManager();
FactoryConfiguration factoryConfig = new FactoryConfiguration();
ToolboxConfiguration toolbox = new ToolboxConfiguration();
// we will have single toolbox instance per application
toolbox.setScope(Scope.APPLICATION);
// next we are directly configuring custom localization tool by
// setting its class name, key, locale property etc.
ToolConfiguration localizationTool = new ToolConfiguration();
localizationTool.setClassname(LocalizationTool.class.getName());
localizationTool.setProperty("locale", LocaleUtility.fromSpecification(locale));
localizationTool.setProperty("bundles", "messages");
// and finally we are adding just configured tool into toolbox
// and creating tool context for this toolbox
toolbox.addTool(localizationTool);
factoryConfig.addToolbox(toolbox);
velocityToolManager.configure(factoryConfig);
ToolContext toolContext = velocityToolManager.createContext();

Our custom localization tool uses custom resourse loader for getting files with messages translations from file system and cachingthem within map object. Sample below shows which code is used for accomplish loading of files and caching them:

File propertiesDir = new File(basedir);
for (File possibleFile : propertiesDir.listFiles()) {
    if (possibleFile.getName().startsWith(FilterUtil.PREFIX) && possibleFile.getName().endsWith(FilterUtil.SUFFIX)) {
	Locale locale = parseLocaleFrom(possibleFile.getName(), FilterUtil.PREFIX);
	getResource().put(locale, getFileSystemResource(possibleFile.getAbsolutePath(), FilterUtil.PREFIX, locale));
	getAvailablelocales().add(locale);
    }
}

This sample of code shows how resource loader searches under the base directory on the file system for possible message properties files and loads them. It iterates over each file, nested to the base directory, and decides if this file is a messages properties. Then, if file is suitable, it parses the locale from its name. And finally, it loads resource bundle for that file and associates it with locale, derived from the file name.

There are also two aspects of functionality, which should be described obligatory.They are related to stroing/retrieving user's locale properties within database. Considering with section Design components behaviour, for persisting locale as admin user's global property and as system default locale property, first it's need to create JDBC connection within database:

connection = DatabaseUpdater.getConnection();

after that we can perform queries through that connection object:

// first we are saving locale as administrative user's property (userId != null) {// first we are saving locale as administrative user's property
if (userId != null) {
    String insert = "insert into user_property (user_id, property, property_value) values (?, 'defaultLocale', ?)";
    statement = connection.prepareStatement(insert);
    statement.setInt(1, userId);
    statement.setString(2, locale);
    if (statement.executeUpdate() != 1) {
        log.warn("Unable to save user locale as admin property.");
    }

}

And finally after all these operations we are closing previously opened session and releasing requested privilege:

finally {
    if (connection != null) {
        try {
            connection.close();
        }
        catch (SQLException e) {
            log.debug("Error while closing the database", e);
        }
    }
}

As for retrieving locale property from database there are some restrictions. We can't use OpenMRS Context class, because spring framework doesn't start till moment of retrieving locales. In fact, we aren't able to use Context.getAdministrationService().executeSQL() method. The solution was to establish JDBC connection and execute raw sql statements through PreparedStatement class. Such like previous sample, first we are establishing connection to database:

Connection connection = DatabaseUpdater.getConnection();

After this, considering with Describe components behaviour section, first we should try to get locale parameter as user's property. Step one is to fetch user from database:

String select = "select user_id from users where system_id = ? or username = ?";
PreparedStatement statement = connection.prepareStatement(select);
statement.setString(1, username);
statement.setString(2, username);
Integer userId = null;
// executing statement ...

and step two is deirect selecting of user's property:

select = "select property_value from user_property where user_id = ? and property = ?";
statement = connection.prepareStatement(select);
statement.setInt(1, userId);
statement.setString(2, OpenmrsConstants.USER_PROPERTY_DEFAULT_LOCALE);
if (statement.execute()) {
    ResultSet results = statement.getResultSet();
    if (results.next())
        currentLocale = results.getString(1);
}

And if user locale property is null we should check default system locale property as follows:

// if locale is still null we should try to retrieve system locale's global property's value
if (currentLocale == null)
    currentLocale = readSystemDefaultLocale(connection);

Even if currentLocale will be null, after this operation we will use OpenmrsConstants.GLOBAL_PROPERTY_DEFAULT_LOCALE_DEFAULT_VALUE as locale parameter. And finally, we should take care about closing the connection to database by calling connection.close().

Description of another aspects of project implementation can be examined by reviewing project sourse code and reading corresponding javadocs.

Performing translations

For performing translations was used Google translation service. All produced translations has been added into existing message properties files under /openmrs-webapp/webapp/src/main/webapp/WEB-INF/ directory. All translations provided as key=value pairs. The keys are described in a way that they are easy to look at the implementation of further translations. Keys of all messages for pages within initial setup wizard is being started with "install.*" prefix. Keys for all error messages for pages within initial setup wizard is being started with "install.error." prefix. The same rule is true for Update wizard. In tis case any key for normal GUI message starts with "update.*"* prefix, for errors during update - with "update.error.*" prefix.

Conclusion

During actual project development the goals, which were originally outlined, have been completed exactly the project proposed with almost no changes at all. Initial time planning, detailed design of components structure, components behaviour, direct implementation of designed components have been completely provided during project development. Custom localization tool, that has been received as main artifact of development, can be used in further for internationalizing Velocity templates within OpenMRS. As for possible improvements in project, new feature, that will allow to localize privileges constants can be added.

Resources

Blog

First presentation

Second presentation

Apache Velocity Developer Guide

Projects

GSOC 2011 Localization of initial setup and update wizard