Sync 2.0 aims to use FHIR to achieve synchronization between a network of nodes. It assumes one master node and multiple slave nodes connected to the master. In this topology, the synchronization can be done in two directions: master to slave and slave to master. Sync 2.0 will support both independently, as one-way sync is being used in the field currently by implementations adopting Sync 1.0. In both models the methods of communication will be generally the same - an atom feed will be read, then data will be retrieved and pushed, preferably using FHIR. If FHIR representations are not available for resources being transmitted, we will fall back to OpenMRS REST representations.
All synchronization will be initiated by slaves, independent from the direction that data is transmitted.
The diagram above gives an overview of a typical Sync network composed of a master server and multiple child nodes synchronizing with it.
Pull from master
In order to keep their data in Sync, Slaves should read data from the master. This will be a three step process:
1) The slave reads through the atom feed exposed by the master and filters out events it is not interested in
2) For events that it is interested in, the data is pulled from master
3) The data is persisted by the slave
Slaves will be reading the feed exposed by master, event by event. The master will expose a complete feed of events - slaves will be in charge of filtering and pulling only the resources they are interested in. When they come across an event that they interpret as requiring synchronization, they will trigger retrieval of the object using the client facade, which will deal with retrieving the object using the correct method of transmission based on the data type - either by delegating to a FHIR client, a REST client or a different client injected by implementation.
The data retrieved through FHIR, will then have to be passed to the FHIR module for saving. FHIR services from the module (example https://github.com/openmrs/openmrs-module-fhir/blob/master/api/src/main/java/org/openmrs/module/fhir/api/impl/PatientServiceImpl.java) will be capable of saving FHIR representations in OpenMRS, so Sync will only need to make use of these services and won't have to convert FHIR to OpenMRS on its own.
Metadata retrieved through the REST API will have to be inserted using conventional methods - best if it directly interfaces with OpenMRS repository classes to insert the metadata into the db.
The slave will keep track of what was synced based on its marker in the masters feed, i.e. the marker should not be moved when a connection error happens. Retries will be controlled by configuration, it should be also possible to make the slave proceed through the feed even if a record fails to synchronize. Since the audit UI will allow manual retry of failed transactions, administrators will be able to resolve conflicts and retry transactions manually.
Pulling data from master can be enabled/disabled independently from pushing data to master in order to allow implementations the possibility to configure one way synchronization.
The graphic below shows the steps required for the pull model:
Push from slave
The push model will be used for pushing data from slaves to the master node. Push is also done in three steps:
1) Read own data feed, filtering out events that don't need to be synchronized with master
2) For events that need synchronization, read them from local endpoints either using FHIR or REST services
3) Push the representation to master
Slaves will publish their own atom feeds, similar to the master. Each slave will read it's own feed and based on what it reads from it, the slave will push the objects to master for synchronization.
After reading the feed, Sync will have to decide whether this is data that should be synchronized with master. If an object in the feed is supposed to get synchronized, the slave will first fetch it from its own FHIR server (or REST if it’s metadata) through localhost and forward it to master. This will allow simplicity with Sync 2.0 code - the mechanism for reading the slaves feed will be similar to the master one. This might also allow multi leveled Sync - since slaves publish their own feeds, they can also act as masters for lower level slaves.
Error handling is similar to the one described above for push.
Pushing data to master can be enabled/disabled independently from pulling the data from master in order to allow implementations the possibility to configure one way synchronization.
The graphic below shows the steps for the push synchronization:
The diagram below illustrates a general overview of the Sync module architecture. Note that components might be simplified and in the real implementation split into multiple pieces. It is shown from the more interesting perspective of a slave. Note that each slave can also act as a Sync master.
Things to note:
- Each slave can also act as a master
- A sync module has two feed reader instances - the local feed reader and the master feed reader. The local feed reader is reading the local feed and triggers a push to master. The master feed reader reads the master feed and triggers a pull from master.
- SyncPushService and SyncPullService put together the business logic for pushing and pulling
- The SyncAuditLogger is responsible for maintaining an audit of Sync operations in the database
- The ClientFacade delegates to either the FHIR client or a REST client, based on the data type and whether a FHIR method is possible. The underlying transfer operation should be transparent to clients.
- SyncPullService uses either FHIR services from the FHIR module or database persistence to save the data locally