Replication and DR options

The purpose of this article

In conjunction with a sound data backup strategy, options regarding Disaster Recovery restore processes must also be considered.

This document aims to provide some insight into two different mechanisms available specific to the PaperTrail application surrounding Replication and DR. Further to this, the pros and cons of each approach are outlined as well as associated cost implications.

As each client environment is unique, this document provides a holistic guide; more detailed analysis of a client’s specific environment may be necessary to make an informed decision on the preferred method.

Option One: PaperTrail – PaperTrail Replication (High Availability)

This option entails continuous 2-way replication between a Master and a Slave server, both running the PaperTrail application. The replication is handled via the PaperTrail application itself and not separately at the DB or file store level.

With replication set in the PaperTrail application, two PaperTrail licenses are required and both servers are required to be online with a continuous HTTP path open between them.

The Master server is currently operational and processing transactions. Once the Slave server has been setup with the pre-requisites and the PaperTrail application, then:

1. All automated import processes on the Master server to be disabled
2. PaperTrail on the current Master server must be taken offline (to obtain a static state of data)
3. DB backup from the Master taken and ingested into the Slave DB
4. File Store Repository and Indexing Data (files on disk) copied from the Master to the Slave
5. DB, File Store and Index replication properties setup on both the Master and Slave systems (PaperTrail offline configuration)
6. Replication increments (2) and offsets configured on both the Master and Slave systems (PaperTrail offline configuration)
7. Configuration changes to the PaperTrail license on the Slave system (host specific license)
8. The Master PaperTrail instance must be brought fully online
9. The Slave PaperTrail instance must be brought fully online
10. Online configuration changes to the Slave PaperTrail instance to change file store server from the hostname of the Master to that of the Slave
11. Restart of the Slave PaperTrail system to allow for the file store changes above
12. Basic testing with dummy data to ensure successful replication between Master and Slave (2-way testing)
13. Re-enable all automated import processes on the Master server

Once setup and running, both instances of the PaperTrail front-end are available as separate interfaces, independent of each other, with an HTTP connection to replicate changes from each side to the other (Master > Slave  and Slave > Master).

Documents created on the Master will assume an even numbering system for the docIDs, while those on the Slave will assume an odd numbering system (due to the increments and offsets configured).

Once an entity (document, node, rule, user, etc) is created, modified or deleted on either system, this transaction (along with associated files and indexes where applicable) are replicated to the other system.

Replication Pros

• Real-time replication ensures for the shortest possible downtime in the event of a system failure
• Data up until the point of failure is available on the replicated system
• Caters for DR testing on the Slave system by simply restricting front-end access to the Master (e.g. firewall rule which still allows only the Slave to replicate). Normal transactions can be carried out on the Slave and will still replicate to the Master to ensure a seamless switchover back to Master (controlled by DNS changes for example)

Replication Cons

• Cost implications associated with two PaperTrail server licenses, including all applicable modules (e.g. workflow, API, eSign). User licenses will be loaded on both systems, but only charged for once
• Cost implications associated with SUA costs on each server
• Does not serve as a failsafe to restore entities (documents, nodes, etc) that were deleted in error on either system. Once executed, the delete transaction is pushed to the replicated system
• Highly complex setup
• High risk involved in the replication of transactions, especially with the level of custom development that is in place (and ongoing) on the environment. This will need to be thoroughly tested in the current state to ensure that transactions are not duplicated with each system sending its own transaction out (e.g. emails, API calls, external DB inserts)
• Any new custom development will need to be tested on an environment setup for replication before it should be considered for deployment to the LIVE systems
• Should either system be offline for a prolonged period of time, PaperTrail’s retry of failed replication transactions will enter a dead state. Replication will then need to be rebuilt from scratch with the associated downtime.
• It is highly advisable that a full DEV system (with Master and Slave replication setup) is available for testing. This will incur additional PaperTrail license and SUA costs
  

Replication Graphical Depiction

Option Two:  Cold Standby System (Replication of File Store As WORM)

This option entails setting up a continuous replication of the primary File Store Repository to a remote host (either LAN or Cloud). The replication is handled from a single online instance of the PaperTrail application (Master).

With this approach, only a single FULL PaperTrail license (server, modules and users) is required as only one instance of the application is online. The Cold Standby system will require a server license only, which is charged as a monthly rental. SUA costs will only apply to a single server, while both will be upgraded. A secondary PaperTrail File Store Repository is configured on the application as a WORM (Write Once, Read Many) File Store.

The Cold Standby system will be prepped (pre-requisites and the PaperTrail application) in isolation, without any impact on the normal running of the Master system. PaperTrail on the Cold Standby system will remain in an offline state.

1. While it is advisable that PaperTrail on the Master is then shutdown and a copy of the File Store Repository is taken in a static state to the Cold Standby system, this is not a necessity (PaperTrail caters for a File Store sync from the primary to the WORM while online).
2. A secondary WORM File Store is configured on PaperTrail on the Master to point to the remote location
3. PaperTrail on the Master needs to be restarted to effect the File Store change
4. File Store sync run from the primary to the WORM store (optional dependent on decision taken in step 1 above)

In the event of DR testing / recovery of accidently deleted data

1. PaperTrail on the Master continues to run uninterrupted with the ongoing WORM File Store replication intact
2. The latest DB backup is decompressed and ingested into the Cold Standby system
3. A PaperTrail license is added to the Cold Standby system and SMTP details are removed to ensure no emails are sent out
4. PaperTrail on the Cold Standby system is brought online
5. The Run As server name for the WORM file store is changed to the hostname of the Cold Standby system
6. All automated import jobs are left as is. These will not run as the Run As server name references the Master system
7. PaperTrail on the Cold Standby system is restarted to effect the File Store change
8. Indexing data is rebuilt on the Cold Standby system
9. The database structure referencing documents and records can be browsed via PaperTrail’s front-end. Previewing and downloading of documents will be catered for by the WORM file store
10. For recovery of accidently deleted data only: Documents can be exported from the Cold Standby system and re-imported back into the Master system via standard PaperTrail functionality on each system
11. PaperTrail on the Cold Standby system shutdown and the temporary license revoked

This option entails setting up a continuous replication of the primary File Store Repository to a remote host (either LAN or Cloud). The replication is handled from a single online instance of the PaperTrail application (Master).

With this approach, only a single FULL PaperTrail license (server, modules and users) is required as only one instance of the application is online. The Cold Standby system will require a server license only, which is charged as a monthly rental. SUA costs will only apply to a single server, while both will be upgraded. A secondary PaperTrail File Store Repository is configured on the application as a WORM (Write Once, Read Many) File Store.

The Cold Standby system will be prepped (pre-requisites and the PaperTrail application) in isolation, without any impact on the normal running of the Master system. PaperTrail on the Cold Standby system will remain in an offline state.

1. While it is advisable that PaperTrail on the Master is then shutdown and a copy of the File Store Repository is taken in a static state to the Cold Standby system, this is not a necessity (PaperTrail caters for a File Store sync from the primary to the WORM while online).
2. A secondary WORM File Store is configured on PaperTrail on the Master to point to the remote location
3. PaperTrail on the Master needs to be restarted to effect the File Store change
4. File Store sync run from the primary to the WORM store (optional dependent on decision taken in step 1 above)

    

In the event of a full failure of the Master system

1. The latest DB backup is decompressed and ingested into the Cold Standby system
2. A PaperTrail license is added to the Cold Standby system
3. PaperTrail on the Cold Standby system is brought online
4. The Run As server name for the WORM file store is changed to the hostname of the Cold Standby system
5. The Run As server name for all automated import jobs is changed to the hostname of the Cold Standby system
6. PaperTrail on the Cold Standby system is restarted to effect the File Store change
7. Indexing data is rebuilt on the Cold Standby system
8. The database structure referencing documents and records can be browsed via PaperTrail’s front-end. Previewing and downloading of documents will be catered for by the WORM file store
9. Access to the PaperTrail front-end can be granted (controlled by DNS changes for example)

Cold Standby Pros

• A server only license is required for the Cold Standby system as a monthly rental cost. No cost implications for users, modules or SUA
• Simple setup that has been thoroughly tested and executed on LIVE production systems by the Egis support team
• No risk associated with duplication of transactions, as only a single instance of PaperTrail (and associated DB) is online at any time
• No need for extensive testing of current and future custom development specific to File Store replication
• Provides a failsafe to restore documents that were deleted in error on the Master. A recurring scheduled DB backup can be set on PaperTrail, which will dump to the primary File Store (with a copy being shipped to the WORM store as well). The DB can be restored to a separate PaperTrail environment and then hooked up to the WORM store to recover documents and import them back into the Master
• Caters for DR testing directly on the remote Cold Standby system without any impact or interruption to normal production activities
• PaperTrail ships with a prebuilt mechanism for connecting Amazon S3 buckets as a remote cloud-based file store
• PaperTrail ships with a prebuilt mechanism to connecting UNC path locations as a remote file store

Cold Standby Cons

• Lead time to get the cold standby system operational is not instantaneous
• Data only up until the point of the last DB backup is available (dependent on the frequency of the DB backup schedule)
• PaperTrail’s disk-based indexing search engine will need to be rebuilt before searching and linking is fully operational
• Should the Master system hardware become available again, a shutdown of PaperTrail and copy of the file store back to the Master is needed. The DB from the Cold Standby system must be backed up and ingested back into the Master

Cold Standby Graphical Depiction – Master Online

Cold Standby Graphical Depiction – Master Offline

This article was written by Sanjay Pather, Support Manager at Egis Software. Sanjay is responsible for managing the ongoing support and maintenance of the PaperTrail application and related dependencies across client instances.

For support: support@egis-software.com | For sales: sales@egis-software.com