In the event of a disaster where the TSM server is lost, the initial step is to rebuild the hardware and operating system to match the previous setup. Once that is complete, the next crucial step is to recreate the TSM instance with the same name as before the crash. To accomplish this, please refer to the guide on “How to create TSM server instance (IBM Spectrum Protect Version 8)” specifically tailored for IBM Spectrum Protect Version 8.
Assuming that you have successfully built the TSM instance, the following steps outline how to perform TSM server database recovery. This procedure is applicable to multiple versions of IBM Spectrum Protect, ranging from version 6 to the latest available version. It serves as a comprehensive guide to guide you through the recovery process.
Note that configuration variables used in the procedure below.
TSM Server Host: my_tsmsvr01
TSM Server Version: Version 8.1.11 IBM Spectrum Protect for Linux/x86_64
TSM BA Client Version: Version 8.1.11
TSM Instance User: tsmsvr01
TSM Instance Group: tsmsrvrs
DSMSEV.OPT file: /opt/tivoli/tsm/tsmsvr01/dsmserv.opt
TSM Device Config File: /mnt/tsmsvr01/tsmdbbackup/devconfig.out
TSM Volume History File: /mnt/tsmsvr01/tsmdbbackup/volhistory.out
TSM DevClass for TSM DB Backup: TSMDBBACKUP
TSM DB Backup Volume: /mnt/tsmsvr01/tsmdbbackup/80048650.DBV
TSM Instance User Home Path: /opt/tivoli/tsmsvr01
TSM Database Path: /opt/tivoli/tsmsvr01/tsmdb/db01
/opt/tivoli/tsmsvr01/tsmdb/db02
/opt/tivoli/tsmsvr01/tsmdb/db03
/opt/tivoli/tsmsvr01/tsmdb/db04
TSM Log Path: /opt/tivoli/tsmsvr01/tsmlog/log01
/opt/tivoli/tsmsvr01/tsmlog/log02
/opt/tivoli/tsmsvr01/tsmlog/log03
/opt/tivoli/tsmsvr01/tsmlog/log04
Prerequisites before starting TSM database recovery
1. Ensure volhistory.out and devconfig.out files safely backed up. devconfig.out file not necessary though because you can always recreate when creating/rebuilding TSM server instance. But volhistory.out is a absolute requirement because it has the volume information where TSM database backup written. 2. Ensure TSM DB Backup volume is present at the right path where volhistory.out file point to. 3. Ensure you have the password ready when you had set up your TSM DB backup devclasss with the protection key enabled. Below is the sample command that you ran when initially setting up TSM server instance. If you haven't set the password using the option protectky=no, then you don't need the password recovering TSM server database.set dbrecovery devclass_name protectkey=yes password=your_password4. Have the SSL certificates ready. Below are the 4 certificate files where they usually reside in TSM home instance user folder. If you just realized that you didn't back up these certificates and can't locate them now, no worries! you can still continue to run TSM server database recovery. But the problem is that new certificate that generated on new TSM server needs to be manually copied and re-certified on EVERY TSM clients for TSM client backup/restore. So it is important to back up these certificates along with devconfig.out and volhistory.out for smooth TSM server database protection.my_tsmsvr01:[tsmsvr01]/opt/tivoli/tsmsvr01>ls -rlt cert.* -rw-------. 1 tsmsvr01 tsmsrvrs 80 Feb 15 23:12 cert.rdb -rw-------. 1 tsmsvr01 tsmsrvrs 80 Feb 15 23:12 cert.crl -rw-------. 1 tsmsvr01 tsmsrvrs 193 Feb 15 23:12 cert.sth -rw-------. 1 tsmsvr01 tsmsrvrs 15080 Mar 3 22:04 cert.kdb
Step 1. Complete TSM server instance build as the same name
1. Complete TSM server instance as the same name of the one that used when TSM database backup completed. Refer to How to create TSM server instance (IBM Spectrum Protect Version 8). 2. Once TSM server instance successfully created, then shutdown TSM server instance.
Step 2. Restore devconfig and volhistory file
1. Ensure that dsmserv.opt file properly locates devconfig.out and volhistory.out.su - tsmsvr01 vi dsmserv.opt VOLUMEHISTORY /mnt/tsmsvr01/tsmdbbackup/volhistory.out DEVCONFIG /mnt/tsmsvr01/tsmdbbackup/devconfig.out2. Restore volhistory.out to the given location as above. Below is the sample output of volhistory.out. Now when TSM database recovery runs, it searches the lines of volhistory.out from the bottom. When it sees the Volume Type: BACKUPFULL then, TSM determines that's the one that needed to start TSM server database recovery. You can freely edit this volhistory.out having the specific volume that you want to use for TSM server database recovery. But ensure the volume information must locate at the very bottom. Just remember that TSM starts to read volhistory.out from the bottom until it finds Volume Type: BACKUPFULL.Operation Date/Time: 2023/03/28 16:57:57 Volume Type: BACKUPFULL Volume Name: "/mnt/tsmsvr01/tsmdbbackup/80047877.DBV" Backup Series: 70 Backup Op: 0 Volume Seq: 100001 Device Class Name: TSMDBBACKUP ************************************************** Operation Date/Time: 2023/03/28 17:10:50 Volume Type: BACKUPFULL Volume Name: "/mnt/tsmsvr01/tsmdbbackup/80048650.DBV" <-- TSM server picks up this volume for DB recovery Backup Series: 71 Backup Op: 0 Volume Seq: 100001 Device Class Name: TSMDBBACKUP **************************************************. <-- End of line of volhistory.out "volhistory.out" 5442L, 177805C3. Locate the TSM DB Backup volume at the given location where the volhistory.out points. From the above example, it's /mnt/tsmsvr01/tsmdbbackup/80048650.DBV. that TSM Server finds and reads for TSM database recovery. 4. Restore devconfig.out file as above path at step# 1. Below is the sample output of devconfig.out. Ensure you have the path available for TSM devclass which accessible with the permission of TSM instance user. However, you can recreate devconfig.out with new TSM instance user as long as it has the same name of devclass that you used when TSM Database backup completed./* Device Configuration */ DEFINE DEVCLASS FILECLASS01 DEVT=FILE FORMAT=DRIVE SHARE=NO MAXCAP=104857600K MOUNTL=30 DIR=/mnt/tsmsvr01/fileclass01 DEFINE DEVCLASS TSMDBBACKUP DEVT=FILE FORMAT=DRIVE SHARE=NO MAXCAP=104857600K MOUNTL=20 DIR=/mnt/tsmsvr01/tsmdbbackup DEFINE SERVER TSMSVR01 COMMMETHOD=TCPIP HLADDRESS=192.168.0.11 LLADDRESS=1500 SERVERPASSWORD=21b809cafc70dd8f9673d19e0fdad0061c SET SERVERNAME TSMSVR01 SET SERVERPASSWORD 211f925a6dbdf29208638111babd62fc79 SERVERBACKUPNODEID 1
Step 3. Restore SSL certificates to the TSM home instance user folder
If you had backed up the SSL certificates, then overwrite them onto the new TSM server home instance folder. This is not required for proceeding TSM Database recovery though. However, without them, new SSL certificates manually re-certified onto every TSM clients for TSM Client Backup/Restore. Below is the list of certificates.my_tsmsvr01:[tsmsvr01]/opt/tivoli/tsmsvr01>ls -rlt cert.* -rw-------. 1 tsmsvr01 tsmsrvrs 80 Feb 15 23:12 cert.rdb -rw-------. 1 tsmsvr01 tsmsrvrs 80 Feb 15 23:12 cert.crl -rw-------. 1 tsmsvr01 tsmsrvrs 193 Feb 15 23:12 cert.sth -rw-------. 1 tsmsvr01 tsmsrvrs 15080 Mar 3 22:04 cert.kdb
Step 4. Clean up TSM server logs (Important!)
If this is in-place and roll-forward TSM database recovery, which means just overwriting TSM database onto the existing one with the latest TSM DB backup, then you don't need to run this step. But if it's a disaster recovery building new server / recreating new TSM instance, then you should run below log cleanup. Otherwise you might end up seeing an error at the end of database recovery process which you waste many hours depending on TSM database size. If that's the case then you should re-create TSM instance AGAIN, clean up the logs and then re-launch TSM server database recovery. It will delay TSM database recovery process big time. ### TSM Log Location in dsmserv.opt ### ACTIVELOGDirectory /opt/tivoli/tsmsvr01/tsmlog/log01 MIRRORLOGDirectory /opt/tivoli/tsmsvr01/tsmlog/log02 ARCHFAILOVERLOGDirectory /opt/tivoli/tsmsvr01/tsmlog/log04 ARCHLOGDirectory /opt/tivoli/tsmsvr01/tsmlog/log03my_tsmsvr01:[tsmsvr01]/opt/tivoli/tsmsvr01>cd tsmlog my_tsmsvr01:[tsmsvr01]/opt/tivoli/tsmsvr01/tsmlog>ls -rlt total 0 drwxr-xr-x. 3 tsmsvr01 tsmsrvrs 22 Feb 15 23:13 log01 drwxr-xr-x. 3 tsmsvr01 tsmsrvrs 22 Feb 15 23:30 log02 drwxr-xr-x. 4 tsmsvr01 tsmsrvrs 38 Feb 15 23:30 log04 drwxr-xr-x. 3 tsmsvr01 tsmsrvrs 22 Feb 15 23:30 log03 my_tsmsvr01:[tsmsvr01]/opt/tivoli/tsmsvr01/tsmlog>rm -rf log01/* my_tsmsvr01:[tsmsvr01]/opt/tivoli/tsmsvr01/tsmlog>rm -rf log02/* my_tsmsvr01:[tsmsvr01]/opt/tivoli/tsmsvr01/tsmlog>rm -rf log03/* my_tsmsvr01:[tsmsvr01]/opt/tivoli/tsmsvr01/tsmlog>rm -rf log04/* my_tsmsvr01:[tsmsvr01]/opt/tivoli/tsmsvr01/tsmlog>ls -rlt total 0 drwxr-xr-x. 2 tsmsvr01 tsmsrvrs 6 Mar 28 17:27 log01 drwxr-xr-x. 2 tsmsvr01 tsmsrvrs 6 Mar 28 17:27 log02 drwxr-xr-x. 2 tsmsvr01 tsmsrvrs 6 Mar 28 17:27 log03 drwxr-xr-x. 2 tsmsvr01 tsmsrvrs 6 Mar 28 17:27 log04 my_tsmsvr01:[tsmsvr01]/opt/tivoli/tsmsvr01/tsmlog>
Step 5. Execute TSM server database recovery
Now, you can run below command to launch TSM database recovery.su - tsmsvr01 /opt/tivoli/tsm/server/bin/dsmserv restore db todate=today totime=now restorekeys=yes password=tsmsvr01_pwd prompt=no
Note! Below is the command in case you configured a different TSM database directory for new TSM instance vs. when it had TSM database backup completed.su - tsmsvr01 /opt/tivoli/tsm/server/bin/dsmserv restore db todate=today totime=now on=/tmp/dbdirs.txt restorekeys=yes password=tsmsvr01_pwd prompt=noIn /tmp/dbdirs.txt, you can just list new path of TSM database directory as below, then TSM will re-format DB2 database to the given location and load TSM DB backup to the folders. Above command often used when changing TSM database path as well. cat /tmp/dbdirs.txt /new_path/tsmdb/db01 /new_path/tsmdb/db02 /new_path/tsmdb/db03 /new_path/tsmdb/db04
Note below TSM server database recovery execution output. Note the line in read. It successfully grabbed the TSM DB Backup Volume and started to read for TSM database recovery.
my_tsmsvr01:[root]/tsm>su - tsmsvr01 my_tsmsvr01:[tsmsvr01]/opt/tivoli/tsmsvr01>/opt/tivoli/tsm/server/bin/dsmserv restore db todate=today totime=now restorekeys=yes password=tsmsvr01_pwd prompt=no ANR7800I DSMSERV generated at 10:26:22 on Nov 3 2020. IBM Spectrum Protect for Linux/x86_64 Version 8, Release 1, Level 11.000 Licensed Materials - Property of IBM (C) Copyright IBM Corporation 1990, 2020. All rights reserved. U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corporation. ANR7801I Subsystem process ID is 3034735. ANR0900I Processing options file /opt/tivoli/tsmsvr01/dsmserv.opt. ANR0010W Unable to open message catalog for language en_US.UTF-8. The default language message catalog will be used. ANR7814I Using instance directory /opt/tivoli/tsmsvr01. ANR3339I Default Label in key data base is TSM Server SelfSigned SHA Key. ANR4726I The ICC support module has been loaded. ANR8598I Outbound SSL Services were loaded. ANR8230I TCP/IP Version 6 driver ready for connection with clients on port 1500. ANR8200I TCP/IP Version 4 driver ready for connection with clients on port 1500. ANR0152I Database manager successfully started. ANR4634I Starting point-in-time database restore to date 03/31/2023 08:38:45 PM. ANR4591I Selected backup series 71 from 03/28/2023 and 05:10:50 PM as best candidate available for restore database processing. ANR4592I Restore database backup series 71 includes eligible operation 0 with volume /mnt/tsmsvr01/tsmdbbackup/80048650.DBV having sequence 100001 and using device class TSMDBBACKUP. ANR4598I Validating database backup information for selected backup series 71 and operation 0 using volume /mnt/tsmsvr01/tsmdbbackup/80048650.DBV. ANR8340I FILE volume /mnt/tsmsvr01/tsmdbbackup/80048650.DBV mounted. ANR1363I Input volume /mnt/tsmsvr01/tsmdbbackup/80048650.DBV opened (sequence number 1). ANR4609I Restore database process found FULL database backup timestamp 20230328171050 from database backup media. ANR1364I Input volume /mnt/tsmsvr01/tsmdbbackup/80048650.DBV closed. ANR3008I Database backup was written using client API: 8.1.9.0. ANR4638I Restore of backup series 71 operation 0 in progress. ANR4897I Database restore of operation 0 will use device class TSMDBBACKUP and attempt to use 1 streams. ANR8592I Session 1 connection is using protocol TLSV12, cipher specification TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, certificate TSM Self-Signed Certificate. ANR0406I Session 1 started for node $$_TSMDBMGR_$$ (DB2/LINUXX8664) (SSL localhost[127.0.0.1]:40312). ANR8340I FILE volume /mnt/tsmsvr01/tsmdbbackup/80048650.DBV mounted. ANR0510I Session 1 opened input volume /mnt/tsmsvr01/tsmdbbackup/80048650.DBV. ANR1363I Input volume /mnt/tsmsvr01/tsmdbbackup/80048650.DBV opened (sequence number 1). ANR4912I Database Restore DB in progress and total bytes transferred 1,073,741,824. ANR4912I Database Restore DB in progress and total bytes transferred 2,147,483,648. ANR4912I Database Restore DB in progress and total bytes transferred 3,221,225,472. ANR0498W Session 2 refused for IOCTSM91 because restore DB is in progress. ANR0498W Session 2 refused for IOCTSM91 because restore DB is in progress. ANR0498W Session 3 refused for IOCTSM91 because restore DB is in progress. ANR0498W Session 3 refused for IOCTSM91 because restore DB is in progress. ANR0498W Session 4 refused for IOCTSM91 because restore DB is in progress. ANR0498W Session 4 refused for IOCTSM91 because restore DB is in progress. ANR0498W Session 5 refused for IOCTSM91 because restore DB is in progress. ANR0498W Session 5 refused for IOCTSM91 because restore DB is in progress. ANR4912I Database Restore DB in progress and total bytes transferred 4,294,967,296. ANR4912I Database Restore DB in progress and total bytes transferred 5,368,709,120. ANR4912I Database Restore DB in progress and total bytes transferred 6,442,450,944. ANR4912I Database Restore DB in progress and total bytes transferred 7,516,192,768. ANR4912I Database Restore DB in progress and total bytes transferred 8,589,934,592. ANR4912I Database Restore DB in progress and total bytes transferred 9,663,676,416. ANR4912I Database Restore DB in progress and total bytes transferred 10,737,418,240. ANR4912I Database Restore DB in progress and total bytes transferred 11,811,160,064. ANR4912I Database Restore DB in progress and total bytes transferred 12,884,901,888. ANR4912I Database Restore DB in progress and total bytes transferred 13,958,643,712. ANR4912I Database Restore DB in progress and total bytes transferred 15,032,385,536. ANR4912I Database Restore DB in progress and total bytes transferred 16,106,127,360. ANR4912I Database Restore DB in progress and total bytes transferred 17,179,869,184. ANR0403I Session 1 ended for node $$_TSMDBMGR_$$ (DB2/LINUXX8664). ANR8592I Session 6 connection is using protocol TLSV12, cipher specification TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, certificate TSM Self-Signed Certificate. ANR0406I Session 6 started for node $$_TSMDBMGR_$$ (DB2/LINUXX8664) (SSL localhost[127.0.0.1]:46366). ANR1364I Input volume /mnt/tsmsvr01/tsmdbbackup/80048650.DBV closed. ANR0514I Session 6 closed volume /mnt/tsmsvr01/tsmdbbackup/80048650.DBV. ANR0403I Session 6 ended for node $$_TSMDBMGR_$$ (DB2/LINUXX8664). ANR8592I Session 7 connection is using protocol TLSV12, cipher specification TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, certificate TSM Self-Signed Certificate. ANR0406I Session 7 started for node $$_TSMDBMGR_$$ (DB2/LINUXX8664) (SSL localhost[127.0.0.1]:36030). ANR0403I Session 7 ended for node $$_TSMDBMGR_$$ (DB2/LINUXX8664). ANR1628I The database manager is using port 51500 for server connections. ANR4635I Point-in-time database restore complete, restore date 03/31/2023 08:38:45 PM. ANR3096I The volumes used to perform this restore operation were successfully recorded in the server volume history. ANR0369I Stopping the database manager because of a server shutdown. my_tsmsvr01:[tsmsvr01]/opt/tivoli/tsmsvr01>
Note the above message in red. You must find the message ANR4635I for validating TSM database recovery process. Now you can safely start TSM server.
Pingback: How to create TSM server instance (IBM Spectrum Protect Version 8) - TechNote101