Restore Globals From Journal Files Using ^JRNRESTO

The InterSystems IRIS ^JRNRESTO routine is used after a database is restored from backup to return it to its state immediately prior to a failure by applying database updates from journal files. This is called a journal restore, and the process of applying the updates is called dejournaling. A journal restore dejournals all journal records created between the creation of the backup and the failure. For example, if the database was backed up early Tuesday morning and crashed on Wednesday afternoon, after you restore the Tuesday backup, you can restore updates from the journal files created on Tuesday and Wednesday.

If sufficient system resources are available, up to 16 dejournaling jobs can perform the updates in parallel within a journal restore operation (see System Requirements for Parallel Dejournaling). This increases the performance of the operation and is called parallel dejournaling. Multiple updaters can update the same database simultaneously, but not the same global, which allows for higher throughput when updates are applied to a small number of databases while ensuring that the data within each global is logically consistent.

Parallel dejournaling is enabled only when the host system has sufficient CPUs to support it and the InterSystems IRIS instance has enough shared memory heap (gmheap) available to allocate for this purpose. In practice, parallel dejournaling will not be used in journal restores on most InterSystems IRIS instances unless gmheap is increased. The number of parallel dejournaling jobs can never exceed the size of the gmheap divided by 200 MB; for example, to support four dejournaling jobs running in parallel, gmheap must be at least 800 MB. (Even if you do not have enough memory available to support parallel dejournaling, dejournaling throughput may improve if you increase the size of the shared memory heap from the default.)

Note:

To change the size of the shared memory heap or gmheap (sometimes known as the shared memory heap or SMH), navigate to the Advanced Memory Setting page (System Administration > Configuration > Additional Settings > Advanced Memory); see Memory and Startup Settings in the “Configuring InterSystems IRIS” chapter in System Administration Guide for more information.

Parallel dejournaling is also used by InterSystems IRIS mirroring; for more information, see Configuring Parallel Dejournaling.

^JRNRESTO restores only to databases whose journal state is Yes at the time of the journal restore. The first time it encounters each database, the routine checks and records its journal state. The restore process skips journal records for databases whose journal state is No. If no databases are marked as being journaled, the routine asks if you wish to terminate the restore; you can then change the database journal state to Yes on specific databases and restart ^JRNRESTO.

Note:

The journal state of a database at the time of restore determines what action is taken; InterSystems IRIS stores nothing in the journal about the current journal state of the database when a given journal record is written. This means that changes to databases whose journal state is Yes are durable, but changes to other databases may not be. InterSystems IRIS ensures physical consistency, but not necessarily application consistency, if transactions involve databases whose journal state is No.

^JRNRESTO lets you make several decisions about the journal restore, as follows:

Restore global updates to databases in the InterSystems IRIS instance in which you are running the routine, or to databases in another InterSystems IRIS instance. You can choose to restore updates for all globals to all databases in the current instance, or to select individual databases in the current or another instance and optionally specify which globals to restore to each.
Restore mirror journal files to a mirrored database (catch up a mirrored database) or to a nonmirrored database. On a mirror member, you are prompted to indicate whether you are catching up a mirrored database, as noted in the following procedure; if so, the procedure is redirected to the MirrorCatchup^ entry point to ^JRNRESTO (see Restore Journal to Mirrored Database Using MirrorCatchup^JRNRESTO).
Apply existing journal filters (see Filter Journal Records Using ^ZJRNFILT) to the restore.
Select a range of journal files to restore from.
Disable journaling of updates during the restore to make the operation faster.
Choose what to do when an error occurs, as follows:
- Continue with the journal restore despite database-related problems, for example, a target database cannot be mounted or an error occurs while applying an update. With this setting, updates to the affected database(s) are skipped; these databases may not be consistent with the other databases following the operation, or may contain inconsistent globals (although the logically consistency of the data within each global is guaranteed) and will need to be recovered separately.
- Abort the journal if an update would have to be skipped due to a database-related problem. With this setting, databases and the globals they contain will be consistent with each other as of the record that caused the restore to be aborted. Parallel dejournaling is disabled with this setting.

Caution:

If you use journal restore scripts based on prompts, you should update the scripts because some prompts may have changed since the last release.

To restore global updates from journal files:

Run the ^JRNRESTO routine in the %SYS namespace, then press <Enter> at the Restore the Journal? prompt to continue.
If you are running the routine on a mirror member, the following prompt is displayed.
```
Catch-up mirrored databases? No =>
```
- If you are restoring mirror journal files to a mirrored database in the same mirror in which the mirror journal files were created, enter yes; the procedure is redirected to the MirrorCatchup^ entry point to ^JRNRESTO (see Restore Journal to Mirrored Database Using MirrorCatchup^JRNRESTO).
- If you are restoring mirror journal files to a nonmirrored database, or are not restoring mirror journal files, enter no or <Enter> and continue to use the procedure described here.
If you have existing journal filters (see Filter Journal Records Using ^ZJRNFILT), specify whether you want to use them:
```
Use current journal filter (ZJRNFILT)? 
Use journal marker filter (MARKER^ZJRNFILT)? 
```
Choose whether you want to restore all journaled globals to all databases in the current InterSystems IRIS instance, or to specify one or more databases and optionally specify which globals to restore to each.
```
Process all journaled globals in all directories? 
```
- Enter Yes if you want to restore all globals to all databases in the current instance.
- Enter No or <Enter> if you want to restore only selected databases in the current or another instance. Then do the following:
  - Indicate whether the journal files were created under a different operating system from that of the current system. This is important because the directory paths you specify for the databases you want to restore must exactly match the paths in the journal files, which are in canonical form. If you respond with No, ^JRNRESTO puts the directory paths you enter into canonical form for the current operating system so they will match those in the journal files. If you respond with Yes, ^JRNRESTO does not canonicalize the paths you enter, because the canonical form in the journal files is different from the canonical form on the current system. In the latter case, you must take care to enter directory paths in the canonical form appropriate to the operating system of the journal files to ensure that they will match.
    For example:
    - if you are working on a Windows system and enter No at this prompt, then enter the path c:\intersystems\iris\mgr\user, ^JRNRESTO automatically canonicalizes this to c:\intersystems\iris\user\ to match journal files created on a Windows system.
    - if you are working on a Unix® system and enter Yes at this prompt because the journal files were created on a Windows system, you must be sure to enter the canonical form of the path, c:\intersystems\iris\mgr\user\, to ensure that it matches the journal files, because ^JRNRESTO cannot canonicalize it for you.
  - Specify each database you want to restore by entering its directory path; this indicates the source database from which the journal records were taken. Press <Enter> at the Redirect to directory prompt to indicate that source and target are the same and restore global updates to the source database. If you are restoring to a different database, for example because you have restored the source database from backup to a different system, enter the directory path of the target database.
    If you are restoring mirror journal files to a nonmirrored database, at the Directory to restore prompt, you can do either of the following:
    - Enter directory path of the source database and then either <Enter> or the directory path of the target nonmirrored database, as described in the foregoing.
    - Enter the full, case-sensitive mirror database name of the source mirrored database, for example,:mirror:JLAP:MIRRORDB, which can be found using the List mirrored databases option on the Mirror Status menu of the ^MIRROR utility, and then specify the directory path of the target nonmirrored database.
    Note:
    
    If you are restoring mirror journal files to a mirrored database, you will not have reached this point in the procedure; see Restore Journal to Mirrored Database Using MirrorCatchup^JRNRESTO.
  - For each database you specify, either confirm that you want to restore updates for all globals or enter one or more globals to restore.
  - When you have entered all the databases, press <Enter> at the Directory to restore prompt, then confirm the list of specified databases and globals.
  For example:
```
Process all journaled globals in all directories? no
Are journal files imported from a different operating system? No => No
 
Directory to restore [? for help]: c:\intersystems\test23\mgr\user\
Redirect to Directory: c:\intersystems\test23\mgr\user\
 => --> c:\intersystems\test23\mgr\user\
Process all globals in c:\intersystems\test23\mgr\user\? No => yes
 
Directory to restore [? for help]: c:\intersystems\test23\mgr\data\
 Redirect to Directory: c:\intersystems\test23\mgr\data\
 => --> c:\intersystems\test23\mgr\data\
Process all globals in c:\intersystems\test23\mgr\data\? No => no

Global ^Survey.LastName
Global ^Survey.ID
Global ^

Directory to restore [? for help]:

Processing globals from the following datasets:
 1. c:\intersystems\test23\mgr\user\   All Globals
 2. c:\intersystems\test23\mgr\data\   Selected Globals:
          ^Survey.LastName
          ^Survey.ID

Specifications correct? Yes =>  Yes
```
  Note:
  
  If you are redirecting two or more databases to the same directory, you must make the same global selection — that is, either enter yes to process all globals, or no and then the same list of globals to process – for all of these databases. If you try to restore multiple databases to a single directory and the global selections are not all the same, the utility gives you the opportunity to either change your database redirection and global selections or cancel the operation.
Specify the journal files to restore from, which should be from the same InterSystems IRIS instance as the source databases you are restoring, by specifying the correct journal history log (see Journal History Log).
- Enter yes or <Enter> at the prompt to use the journal history log of the current InterSystems IRIS instance to identify the journal files to process. For example, if you entered yes at the Process all journaled globals in all directories? prompt at the start of the process, enter yes here to restore all databases in the current instance from the current instance’s journal files.
- If you entered no at the Process all journaled globals in all directories? prompt and then specified databases in another InterSystems IRIS instance, enter no here to specify the journal history log and journal file directory path of that instance, or files copied from that instance, so that the databases can be restored from that instance’s journal files.
  
  Important:
  
  If you are using a journal history log from another InterSystems IRIS instance, you must use a copy of the file, not the actual log.
  
  For example,
```
If you have a copy of the journal history log file from the 
instance where the journal files were created, enter its full path below;
otherwise, press ENTER and continue.
Journal history log: c:\InterSystems\IRIS23_journals\journal.log

Specify the location of the journal files to be processed
Directory of the journal files:  c:\InterSystems\IRIS23_journals\journal\
Directory of the journal files: 
```

Specify the range of journal files you want to process. Bear in mind the following:

If InterSystems IRIS switched to multiple journal files since the restored backup, you must restore the journal files in order from the oldest to the most recent. For example, if you have three journal files to restore, 20230214.001, 20230214.002, and 20230215.001, you must restore them in the following order:
20230214.001
20230214.002
20230215.001
When you back up with online backup, information about the oldest journal file required for transaction rollback during restore is displayed at the beginning of the third and final pass and stored in the backup log. See Backup and Restore for more information.

Respond to the prompts as follows:

If you entered yes at the earlier prompt about whether this instance creates journal files, or answered no and then specified the journal history log and journal file location of another instance, you can enter the pathnames of the first and last journal files to process. You can also enter ? at either prompt to see a numbered list of the files in the specified location, then enter the numbers of the files, for example:

Specify range of files to process
Enter ? for a list of journal files to select the first and last files from
First file to process:  ?
 
1) c:\intersystems\iris2\mgr\journal\20230212.001
2) c:\intersystems\iris2\mgr\journal\20230213.001
3) c:\intersystems\iris2\mgr\journal\20230214.001
4) c:\intersystems\iris2\mgr\journal\20230214.002
5) c:\intersystems\iris2\mgr\journal\20230215.001
6) c:\intersystems\iris2\mgr\journal\20230216.001
7) c:\intersystems\iris2\mgr\journal\20230217.001
8) c:\intersystems\iris2\mgr\journal\20230217.002

First file to process:  5 c:\intersystems\iris2\mgr\journal\20230215.001
Final file to process:
  c:\intersystems\20231316mar14\mgr\journal\20230217.002 => 

Prompt for name of the next file to process? No => no

If you entered no at the earlier prompt about whether this instance creates journal files and did not specify a journal history log, processing continues with prompts that attempt to identify the specific journal files you want to process. For example:

Journal history log:
Specify range of files to process (names in YYYYMMDD.NNN format)
 
from:     <20230212.001> [?] => 20230215.001
 
through:  <20230217.001> [?] => 20230217.002
 
Prompt for name of the next file to process? No => no
 
Provide or confirm the following configuration settings:
 
Journal File Prefix: [?] =>
 
Files to dejournal will be looked for in:
     c:\intersystems\iris\mgr\journal\
in addition to any directories you are going to specify below, UNLESS
you enter a minus sign ('-' without quotes) at the prompt below,
in which case ONLY directories given subsequently will be searched

Directory to search: {return when done} -
     [Directory search list is emptied]
Directory to search: {return when done} c:\intersystems\iris2\mgr\journal
Directory to search: {return when done}
Here is a list of directories in the order they will be searched for files:
     c:\intersystems\iris2\mgr\journal\

Process the journal files:

Prompt for name of the next file to process? No => No 
The following actions will be performed if you answer YES below:
 
* Listing journal files in the order they will be processed
* Checking for any missing journal file on the list ("a broken chain")
 
The basic assumption is that the files to be processed are all
currently accessible. If that is not the case, e.g., if you plan to
load journal files from tapes on demand, you should answer NO below.
Check for missing journal files? Yes => Yes

If one or more journal files within the range you specified are missing, you are given the opportunity to abort the operation. If you do not, or if no files are missing, the process proceeds with an opportunity to check journal integrity before starting the restore:

Journal files in the order they will be processed:
1. c:\intersystems\iris2\mgr\journal\20230215.001
2. c:\intersystems\iris2\mgr\journal\20230216.001
3. c:\intersystems\iris2\mgr\journal\20230217.001
4. c:\intersystems\iris2\mgr\journal\20230217.002

While the actual journal restore will detect a journal integrity problem
when running into it, you have the option to check the integrity now
before performing the journal restore. The integrity checker works by
scanning journal files, which may take a while depending on file sizes.
Check journal integrity? No => No

If the current journal file is included in the restore, you must switch journaling to another file, and are prompted to do so:

The journal restore includes the current journal file.
You cannot do that unless you stop journaling or switch
     journaling to another file.
Do you want to switch journaling? Yes => yes
Journaling switched to c:\intersystems\iris2\mgr\journal\20230217.003

Next, choose whether to disable journaling of updates during the restore to make the operation faster.
```
You may disable journaling of updates for faster restore for all
databases other than mirrored databases. 
Do you want to disable journaling the updates? Yes => yes
Updates will NOT be journaled
```
Important:

If you do not disable journaling of updates during the restore, parallel dejournaling will not be used to increase performance, as described at the beginning of this section.
If journaling is disabled but database updates continue, you cannot use the last good journals to do a manual restore unless you can assure either of the following:
- You know exactly what will be updated and can control what is restored to the satisfaction of the application.
- You have restored the database(s) involved from the last backup and accept that after applying the journals you will have lost the data written when journaling was off.
InterSystems recommends that you run the following commands after completing the journal restore under these circumstances to verify that the object IDs are not out of sync; only IDs that are found to be out of sync are reported in the array, errors:
```
Do CheckIDCounters^%apiOBJ(.errors)
zwrite errors
```

After confirming or changing the default behavior on error for the restore job, confirm the restore to begin:

Before we job off restore daemons, you may tailor the behavior of a
restore daemon in certain events by choosing from the options below:
 
     DEFAULT:    Continue despite database-related problems (e.g., a target
     database cannot be mounted, error applying an update, etc.), skipping
     updates to that database. Affected database(s) may not be self-consistent
     and will need to be recovered separately
 
     ALTERNATE:  Abort if an update would have to be skipped due to a
     database-related problem (e.g., a target database cannot be mounted,
     error applying an update, etc.). Databases will be left in a
     self-consistent state as of the record that caused the restore to be
     aborted. Parallel dejournaling will be disabled with this setting  

     DEFAULT:    Abort if an update would have to be skipped due to a
     journal-related problem (e.g., journal corruption, some cases of missing
     journal files, etc.)
 
     ALTERNATE:  Continue despite journal-related problems (e.g., journal
     corruption, some missing journal files, etc.), skipping affected updates
 
Would you like to change the default actions? No => No
  
Start the restore? Yes =>

Important:

If you choose to abort due to database-related problems, parallel dejournaling will not be used to increase performance, as described at the beginning of this section.

The progress of the journal restore is displayed at intervals, and when the job is complete a list of databases updated by the restore is displayed:

c:\MyIRIS1\mgr\journal\20230406.001
 35.73%  70.61% 100.00%
c:\MyIRIS1\mgr\journal\20230406.002
 35.73%  70.61% 100.00%
c:\MyIRIS1\mgr\journal\20230406.003
 100.00%
[Journal restore completed at 20230407 02:25:31]
 
The following databases have been updated:

1. c:\MyIRIS1\mgr\source22\
2. c:\MyIRIS1\mgr\source23\
3. c:\MyIRIS1\mgr\irislocaldata\
4. c:\MyIRIS1\mgr\irislib\
5. c:\MyIRIS1\mgr\iristemp\

The following databases have been skipped:

1. /bench/user/InterSystems/IRIS/162/
2. /scratch1/user/InterSystems/IRIS/p750.162/mgr/
3. /scratch1/user/InterSystems/IRIS/p750.162/mgr/irislocaldata/
4. /scratch1/user/InterSystems/IRIS/p750.162/mgr/irislib/
5. /scratch1/user/InterSystems/IRIS/p750.162/mgr/iristemp/
6. /scratch1/user/InterSystems/IRIS/p750.162/mgr/user/

If there are any open transactions, you will be prompted at the end of the restore to roll back the incomplete transactions:
```
Rollback incomplete transactions? No =>
```
This prompt will only appear if there are any incomplete transactions in the any of the databases being restored. See Rolling Back Incomplete Transactions for more information.

Rolling Back Incomplete Transactions

Restoring the journal will also prompt you to roll back incomplete transactions if there are any. Ensure that users have completed all transactions so that the restore does not attempt to roll back active processes.

To ensure that transactions are all complete before you restore your backup and clear the journal file, InterSystems strongly recommends the following:

If you need to roll back transactions for your own process, the process must halt or use the TROLLBACK command.
If you need to roll back transactions system-wide, shut down InterSystems IRIS and restart it to ensure that no users are on the system.

Restoring Mirror Journal Files

You can restore mirror journal files to either mirrored or nonmirrored databases. If you are restoring to a mirrored database, see step 2 of the procedure in Restore Globals From Journal Files Using ^JRNRESTO and Restore Journal to Mirrored Database Using MirrorCatchup^JRNRESTO. If you are restoring to a nonmirrored database, see step 4 of the procedure in Restore Globals From Journal Files Using ^JRNRESTO.

Filter Journal Records Using ^ZJRNFILT

InterSystems provides a journal filter mechanism to manipulate the journal file. The journal filter program is a user-written routine called ^ZJRNFILT whose format is shown below. This is called by the InterSystems IRIS journal restore program, ^JRNRESTO, and ensures that only selected records are restored. Create the ^ZJRNFILT routine using the following format:

ZJRNFILT(jidsys,dir,glo,type,restmode,addr,time)

Argument	Type	Description
jidsys	input	Two components separated by a comma: jid (job ID) and remsysid (for ECP only). Pass jidsys to the %SYS.Journal.Record.GetRealPIDSYSinFilterOpens in a new tab method (as shown in the “^ZJRNFILT Examples” below) to identify the PID (process ID) that generated the journal.
dir	input	Full pathname of the directory containing the IRIS.DAT file to be restored, as specified in the journal record.
glo	input	Global in journal record.
type	input	Command type in journal record (S for Set, K for Kill).
addr	input	Address of the journal record.
time	input	Time stamp of the record (in $horolog format). This is the time the journal buffer is created, not when the Set or Kill operation occurs, so it represents the earliest this particular operation could have happened.
restmode	output	0 - do not restore record. 1 - restore record.

^ZJRNFILT Considerations

Consider the following when using ^ZJRNFILT:

If the startup routine (^STU) calls ^JRNRESTO, it does not call the filter routine under any circumstances.
Journal restore only calls the journal filter (^ZJRNFILT) if it exists. If it does exist, the restore procedure prompts you to confirm the use of the filter in the restore process.
If you answer yes to use the journal filter, for every record in the journal file to restore, the routine calls the journal filter ^ZJRNFILT with the indicated input arguments to determine whether to restore the current record.
You can use any logic in your ^ZJRNFILT routine to determine whether or not to restore the record. Return confirmation through the output restmode argument.
If you are using the directory name, dir, in the ^ZJRNFILT routine logic, specify the full directory pathname.
The entire global reference is passed to ^ZJRNFILT for use in program logic.
When the journal restore process completes, it prompts you to confirm whether to rename the ^ZJRNFILT routine or delete it. If you choose to rename the filter, the utility renames it ^XJRNFILT and deletes the original ^ZJRNFILT.
The restore process aborts with an appropriate error message if any errors occur in the ^ZJRNFILT routine.

^ZJRNFILT Examples

Two globals, ^ABC and ^XYZ, are journaled. While journaling is turned on, the following code is executed, and the journal file records the Set and Kill operations for these globals:

 For I=1:1:500 Set ^ABC(I)=""
 For I=1:1:500 Set ^XYZ(I)=""
 For I=1:1:100 Kill ^ABC(I)

To restore all records for ^ABC only, the ^ZJRNFILT routine looks like this:

ZJRNFILT(jidsys,dir,glo,type,restmode,addr,time)    /*Filter*/
 Set restmode=1                                  /*Return 1 for restore*/
 If glo["XYZ" Set restmode=0                     /*except when it is ^XYZ*/
 Quit
 ;

To restore all records except the kill on ^ABC, the ^ZJRNFILT routine looks like this:

ZJRNFILT(jidsys,dir,glo,type,restmode,addr,time)    /*Filter*/
 Set restmode=1                                  /*Return 1 for restore*/
 If glo["^ABC",type="K" Set restmode=0           /*except if a kill on ^ABC*/
 Quit
 ;

In some cases (for example, when the jid is a PID or on a mirror member), remsysid is not the actual ECP system ID. In these cases, use the %SYS.Journal.Record.GetRealPIDSYSinFilterOpens in a new tab method to return the real ECP system ID as well as the real PID.
To pull the real PID (and ECP system PID) in a filter, the ^ZJRNFILT routine looks like this:
```
ZJRNFILT(jidsys,dir,glo,type,restmode,addr,time) ;
 SET restmode=0 ;test only
 SET pid=##class(%SYS.Journal.Record).GetRealPIDSYSinFilter(jidsys,.ecpsysid)
 DO ##class(%SYS.System).WriteToConsoleLog($SELECT(pid="":"jid="_+jidsys,1:"pid="_pid)_",ecpsysid="_ecpsysid)
 QUIT
```

To restore all records after a specific time, the ^ZJRNFILT routine looks like this:

ZJRNFILT(jidsys,dir,glo,type,restmode,addr,time) ;
 New zdh 
 Set zdh=$zdatetimeh("08/14/2015 14:18:31") ;in $H format as variable 'time'
 If time>zdh Set restmode=1 Quit
 If time<zdh Set restmode=0 Quit
 If $p(time,",",2)<$p(zdh,",",2) {
  Set restmode=0
 } Else {
    Set restmode=1
 }
 Quit

Display Journal Records Using ^JRNDUMP

Switch Journaling Directories Using SWDIR^JOURNAL