Sync Module

Owned by Former user (Deleted)
Last updated: Dec 16, 2017 by Fawwaz YusranVersion comment
9 min read

Overview

Synchronization is needed when you have multiple sites using OpenMRS with separate databases and you want them to copy data to each other. In this situation, one site has to act as the "Parent", and one acts as the "Child". The Parent is always the computer or site that is most central. Multiple child sites send their data up to the parent site for dissemination. Children/parent sites can be set up in a multi hierarchy way. (eg. a parent server can have a parent server). The parent/child servers can be connected by a network continuously, connected occasionally, or not connected at all.

The Sync Module works by keeping track of all changes going through the API into the database. These changes are then replayed on the other server(s). The changes are stripped of all primary key references. Instead, UUIDs are used to compare and reference other data. Because of this, the parent/child servers must be exact duplicates when sync is first started.

This module uses code from the Data Synchronization Project and should be installed on OpenMRS v1.8.3 or greater.

Download and Installation

Updating The Sync Module to a New Version

The sync module is a Mandatory Module and cannot be stopped or unloaded without stopping OpenMRS. This prevents any data loss for connected parent/child servers from occurring. However, it means that there is no way to remove the module to install a new one in the traditional fashion.

  1. Download the latest version of the omod
  2. Option #1:
    1. Visit the Manage Modules admin page and upload the module with the "Upgrade An Existing Module" form on the right side of the page
  3. Option #2:
    1. Visit the Module Properties admin page (/openmrs/admin/modules/moduleProperties.form) to find where your modules are stored on the server.
    2. Delete the old sync-0.1.omod file
    3. Copy in the new sync-0.2.omod file
    4. Restart openmrs and/or tomcat

Uninstalling the Sync Module

The sync module is a Mandatory Module and cannot be stopped or unloaded without stopping OpenMRS. This prevents any data loss for connected parent/child servers from occurring. However, it means that there is no way to delete or uninstall the module from the web interface.

  1. Set the sync.mandatory module to false.
  2. Visit the Module Properties admin page (/openmrs/admin/modules/moduleProperties.form) to find where your modules are stored on the server filesystem.
  3. Delete the sync-*.omod file
  4. Restart openmrs and/or tomcat

To clean up the database, these statements need to be executed. Be careful as you are also deleting all the sync history and can't resume your work with the Sync Module later on.

drop table sync_server_record;
drop table sync_server_class;
drop table sync_server;
drop table sync_class;
drop table sync_record;
drop table sync_import;
delete from global_property where property like 'sync.%';

Additionally you should stop and remove the Scheduled Tasks "Cleanup Old Sync Records" and "master Data Synchronization" (if the automatic sync via web is configured).

Upgrading OpenMRS when using the Sync Module

When upgrading OpenMRS, occasionally data is modified that is key to sync (a uuid column). This is done outside the API, so care must be taken to update the child sites correctly.

This is the general process you should follow whenever upgrading openmrs

  1. Make sure all sites are sync'd to their parents and no data entry is happening on any parent/child server
  2. Go through the upgrade process on the parent server. (Download+install the new war file, view the webapp and run all database updates)
  3. Immediately visit "Sync --> Upgrade Scripts" on the administration screen
  4. On that form, if upgrading from 1.5, choose 1.5.* and click the Download button (if your old version is not there, no updates are needed)
  5. Run the sql script on each child database (mysql -u -p -e"source upgrade.sql" openmrs)
  6. Go through the upgrade process on each child server (similar to the second step)

WARNINGS!

Direct Database Updates

Once the sync module is installed, all database updates should be done through the openmrs api. Direct sql on the database should not be used because those changes will not cascade to the parent/child servers.

Instead use the groovy module to make calls on the OpenMRS API to change the database.

(TODO: Add examples here)

Conflict Handling/Resolution

The initial version of the sync module provides no means for conflict handling or resolution: it simply operates on a last-in-wins model. The assumption is that patient-level conflicts are rare in practical use, and therefore conflict handling is not a "must-have" feature for a version 1.0 of the system. It is recommended that the Sync module, as currently designed, NOT be used in situations where frequent conflicts are expected (aka concurrent patient editing).

Using Sync with Infopath

It is strongly recommended that you don't depend on sync to propagate changes to your infopath forms. 

Infopath XSNs are .cab files that contain a whole bunch of files inside of them.  Some of these files have the hard-coded paths of the server that one was working on when modifying the form.  The problem is that these hard-coded strings then get propagated to the other servers through sync (when it's working).   In the past, we've seen forms from one site being submitted to another site, which caused all kinds of problems.

It is recommended to block synchronization of org.openmrs.module.formentry, and do all XSN manipulation by hand on each server. 

Alert and AlertRecipient

These objects are supported in Sync Module Beta 2 + and with OpenMRS v1.5.3+, v.1.6.2+, and 1.7+. Any other version and you will have to "ignore" the class "org.openmrs.notification.AlertRecipient" from both sending and receiving in order to avoid errors.

Compatibility with other Modules

Formentry Module

If using the Formentry Module 5.x and below with the Sync Module you must take care with how the concepts are developed. Formentry still depends on internal concept.concept_id values being the same on all servers. The sync module cannot guarantee that to be the case. So all concept/form development must happen on one and only one server in your sync network.

See full note on Formentry Module wiki page.

History of the Sync Project

The sync project was the brain child of Christian Allen in 2007 while he was working for Partner's in Health. PIH had multiple sites that were not continuously connected by internet across the Rwandan countryside. Transporting paper forms 4-5 hours on a daily or even weekly basis was not working. His vision of a bidirectional asynchronous database synchronization technique lived and developed in a branch or two for several years. In mid 2009 it was clear that sync was not going to be a part of core code going forward. We made changes to core that were incorporated into the OpenMRS 1.5 release to enable sync to be a module. We then ported the code from the branch into the sync module to allow for maximum flexibility and ease of use.

Release Notes

  • 1.1
    • [SYNC-54] - Add Javascript Refresh to Overview page in Sync Module
    • [SYNC-65] - Add link to Scheduler in Sync Module
    • [SYNC-194] - null pointer hit when onFlushDirty is exectued before afterTransactionBegin in interceptor
    • [SYNC-242] - Fix failing sync unit tests
    • [SYNC-245] - From History of Changes page, unable to reset a record as a new sync record if any of the child server records are in the "waiting to synchronize" state
    • [SYNC-247] - Configure sync poms to allow testing against multiple versions of Openmrs
    • [SYNC-253] - failing unit test against 1.9.0-alpha: shouldAddObsWithEncounter(org.openmrs.module.sync.SyncEncounterTest)
    • [SYNC-254] - failing unit test against 1.9.0-alpha: shouldMergePatients(org.openmrs.module.sync.SyncPatientTest)
    • [SYNC-255] - failing unit test against 1.9.0-alpha: shouldNotGenerateSystemIdIfGPNotDefined(org.openmrs.module.sync.SyncUserTest)
    • [SYNC-272] - History of Changes: Records per page should not set "firstRecordId" if not previously set
    • [SYNC-274] - When adding classes to exclude/include, trim off any trailing spaces
    • [SYNC-276] - SyncIngestService should also not update records with state COMMITTED_AND_CONFIRMATION_SENT
    • [SYNC-278] - Add index to original_uuid column in sync_record table
    • [SYNC-279] - Index sync_record.timestamp, sync_server_record.server_id & .state
    • [SYNC-280] - Voided PersonAttribute get deleted when syncing
    • [SYNC-282] - Problem adding existing concept to an existing concept set
    • [SYNC-284] - failing unit test against 1.9.1: shouldFindUuidsOnConcepts(org.openmrs.module.sync.SyncORUR01HandlerTest)
    • [SYNC-287] - Sync module should prevent OpenMRS Core from persisting changes if no sync record is created
    • [SYNC-302] - Fix failing unit test: shouldDeletePersonAttributeType(org.openmrs.module.sync.SyncOnDeleteTest)
    • [SYNC-304] - Sync module fails to load on 1.9.3 standalone
    • [SYNC-306] - If the only changes that occur to an object are within a collection property, sync fails to detect and record this change
    • [SYNC-307] - Sync fails to save sets and maps of non-openmrs objects if the owning object has no updates
    • [SYNC-308] - History of changes page does not correctly use configured setting for number of records per page
    • [SYNC-311] - History page throws ELException on Tomcat7
    • [SYNC-323] - Improve performance of "lastRecord" and "previousRecord"
    • [SYNC-324] - Stacktrace from SyncInterceptor when trying to save user properties
    • [SYNC-325] - Sync Records are lost if multiple transactions run on the same thread
    • [SYNC-265] - liquibase changeset 2011121-1030k needs special handling in sync context
    • [SYNC-277] - Create SyncUtil.isFinalState(state) method
    • [SYNC-288] - Add sync record id to the history of changes page
    • [SYNC-289] - Show current time on the overview page
    • [SYNC-293] - Add quick link on history page to "Most recent all committed"
    • [SYNC-294] - Ensure outgoing sync error details are properly logged and available to the administrator
    • [SYNC-305] - Merge sprint201303 branch into master
    • [SYNC-313] - Add ability to test email configuration

  • 1.0.7
    • SYNC-290 - Sync Statistics page throws an error upon loading
    • SYNC-298 - Investigate and fix failing unit test: shouldSaveObsAndProtectObsIdInVoidReason(org.openmrs.module.sync.SyncObsTest)
    • SYNC-303 - Fix project layout to meet current best practices
  • 1.0.6
    • SYNC-286 - If you click on the "Maintenance" tab for Sync Administration, the repeat interval on the task to clean the records gets set to sec
  • 1.0.5
    • SYNC-250 - Sync module should be able to handle properties that are enums
    • SYNC-257 - Sync module doesn't build against 1.10-SNAPSHOT
    • SYNC-267 - Sync should explicitly set character encoding to UTF-8 when creating and reading sync transmissions
    • SYNC-283 - Problem syncing changes to concept names
    • SYNC-266 - Add unique constraint to sync_import.uuid column
  • 1.0.4
    • SYNC-283: Problem syncing changes to Concept names
    • SYNC-242: Fix failing sync unit tests
  • 1.0.3
    • SYNC-280: Voided PersonAttribute gets deleted while syncing
  • 1.0.2
  • 1.0.1
  • 1.0
    • Fixed many bugs and added a few more features
    • Only known sync'ing issue is with formentry xsn objects. See note above.
  • 0.972
    • Fixed SQL LEVEL ACCESS privilege requirement SYNC-179
    • Fixed records marked as Do Not Process from holding up the queue SYNC-193
    • Sync'ing concepts first in the list SYNC-180
    • Removing the special-cased hard copy of Concept.conceptId and depending solely on uuid SYNC-160
    • Added a socket timeout setting SYNC-169
    • Fixed PersonAttribute sync'ing SYNC-177
    • Fixed hibernate error with new records SYNC-175
    • Made configuration timeout configurable SYNC-180
    • Fixed sync'ing of concept answers SYNC-174
  • 0.94
    • Fixed failing unit tests and flaky logic when creating new patients
    • Fixed sync'ing of class objects - SYNC-172
  • 0.93
    • Fixed Sync statistics page causing OOM when there are a lot of sync records - SYNC-165
  • 0.92
    • Fixed sync'ing of object attributes marked as 'text' SYNC-171
  • 0.91
  • 0.9 (official "RC 1" release)
    • Fixed syncing of concepts so that it is compatible with metadata sharing module (i.e. concept_ids are preserved)   SYNC-160, SYNC-166
    • Added ability to sync persistent maps (this enables sync-ing of person attributes) - SYNC-59, SYNC-164
    • Added Sync In Progress indicator - SYNC-23
    • Better logging of errors for FAILED records - SYNC-58
    • (and more) 
  • 0.8 (official "beta 2" release)
    • Exclude certain types (Scheduler, hl7, global property) from sync'ing. Adding package exclusions - #2352
    • PersonAttirbutes with empty values fixed - #2360
    • Indeterminate PatientIdentifier errors - #1535
    • AlertRecipient can be sync'd now - #2233
    • Cohort members can be sync'd - #2362
    • Sync setup on 1.5.x was failing - #2389
    • (and more)
  • 0.7 (official "beta" release)
    • Added Antoine's new sync admin functionality and ability to generate/send/receive database backup over the web - #547
    • Show number of sync incoming sync items - #2014
    • Fixed receiving of LoginCredential objects - #2236
    • Made rebuild of xsn conditional on xsn existence - #1440
    • Fixed display of error translations - #612
    • Added more description around received items on status page - #602
    • KNOWN ISSUE: AlertRecipient cannot be received - #2233
  • 0.6.1 (official "alpha" release)
    • Changed the default logging so that there isn't as much noise in the server log files
  • 0.6
    • Fixed concept word rebuilding when new concepts are added. This allows new concepts to be searched/found at remote sites after a new concept was created on central
  • 0.5
    • Added ability to generate/download upgrade scripts between OpenMRS 1.5 & 1.6
    • Minor Sync API cleanup - archive:TRAC-2029
  • 0.4 (Initial public pre-release)
    • Fixed saving of non foreign keyed links to primary key columns in voided obs, person attribute and attribute types
    • Changed all classloader calls to go through the openmrs classloader to enable sync'ing of module objects
  • 0.3
    • Fixed offline file transfer to work in bidirectional fashion
    • Fixed cohort sync'ing
  • 0.2
    • Changed classes to shorter sync name
    • Added ability for module classes to be sync'ed
    • Fixed user id generation
    • Reorganized administration pages
  • 0.1
    • Initial release
    • Contains all functionality from sync bidir branch

Resources

For development purposes, see the Active Sync Module tickets