Online ACB library cut over – IMS Contention Issue

Problem:

Planning for high availability change. There was only a new PSB to be defined as part of a release. NEW PSB can be defined in IMS region dynamically. During a high available release, the PSB gen was generated for the NEW PSB alone, but the ACB was built with BUILD ALL.  So during modify prepare and commit it was giving lots of the contentions as it was trying to refresh all ACB members.

Reason:

The modify commit will not be successful as all members of the ACBLIB was considered as changed.  Only we had 2 options, bring down all MPP region and IMS Databases down to complete the online ACB flip or reset everything and fix the ACB library. We decided to do the second one as it was better option in this case.

Solution:

Copy all the members from active ACBLIB library to the ACBLIB and inactive ACBLIB library. To bring back the ACB library to original status and then redo the ACB build with build for the new PSB alone.

Lessons Learnt:

ACB built should be for the only the new or changed member alone for a high availability release. ACB Build all is need only during IMS version upgrade or a major outage release when many members are impacted.

IMS Fast Path - Dataset Definitions Inconsistent

Problem:

Production application job and DBA Fast Area health check job failed. A DEDB area was inaccessible to the requested service when the application program issued a database request or when the application program reached a commit point. The AREA was stopped due to inconsistent dataset definitions.
+DATABASE=DBNAME FORMAT=DEDB ACTION=FH=IMS AREAS=N/A

REASON CODE=28 AREA=NAMExxxx DBD AND DATA SET DEFINITIONS INCONSISTENT IMXX

AREA=NAMExxxx DD=FCALxxxx CLOSED IMXX

DATASET NAMExxxx SUCCESSFULLY DEALLOCATED IMXX
AREA=NAMExxxx STOPPED IMXX

DFS554A JOBNAMEX 00002 STEP01 PGMNAMEX(2) 000,4000 PSB 2013/104 1:00:35

Reason:

REORG/RESIZE was cancelled for last week due to IPL and Business requirement in the last movement. REORG/RESIZE setup completed but schedule flow cancelled spotlight the bug in the REORG setup tool. A Fast path Area selected to be RESIZED last weekend was picked up by the DBDGEN process during this week RESIZE Flow, even though a different AREA of different database was selected to be RESIZED this weekend after deleting last week setup.

The 2 AREA’s of 2 different Fast path database were selected for RESIZE for DBDGEN process but REORG after DBD change did complete for only one AREA.

Even the ACBLIB flip in online region completed after the RESIZE/REORG flow completed.


Fix:

The impacted AREA was REORG/RESIZED by unloading the AREA using the inactive ACBLIB . Delete/define and load the AREA using the active ACBLIB. The key is to use the inactive ACBLIB during unload as otherwise the job will fail with message “DMAC inconsistent with DMB”. Run Pointer checker to confirm there is no issue after REORG and started the AREA fixed the problem.

Lesson Learnt: 

Validate the DBDGEN after the REORG/RESIZE setup for correct list of AREA’s.
The key to fix the problem if occurs, is to use the inactive ACBLIB during unload as otherwise the job will fail with message “DMAC inconsistent with DMB”. Validate the delete /define to match the RESIZE Requested.

IMS Version 12 Upgrade issue


After IMS version 12 upgrade from version 10, Version 10 BMC Image copy became incompatible with IBM IMS recovery. However, BMC recovery works fine.



Reason:

As per IBM IMS standard the first byte of Imagecopy dataset should be as follows.

‘05’ – Version 9

‘06’- Version 10

‘07’ – Version 11

‘08’ – Version 12

IMS version 10, Image copy taken by IBM IMS image copy uses ’06’ as first byte whereas BMC Image copy uses ‘05’ as the first byte.  IBM version 12 Recovery is compatible only with version 11 and Version10.  So IBM recovery failed to recover from BMC version 10 Image copy as it had wrong header value of ‘05’ instead of ‘06’.

Work Around: BMC recovery version 4.5 that runs on IMS V12 is compatible with its BMC Image copy of version 10

Fix:

PTF named BPQ6812 (by BMC support) which will fix our invalid BMC Image copy header issue.

IMS-Backout required for successfully completed PSB

Problem :

Found that the Application team wanted to do backout for a successfully completed PSB XXXXXX and rerun the step XXXXX of the job, as she wanted to process some records that were rejected by the program.

Solution:
Key points to backout successfully completed IMS Batch job.
1) The DBRC parameter has to be coded as 'C'
2) IMSLOGR should be the log from the program
3) IEFRDER should be the log for backout job
4) BYPASS DD DUMMY
5) SYSIN coded with BYPASS LOGVER

After backout run pointer checker was run against the affected databases and it did not have any issues. The SE restarted the process and it ran as expected.

IMS RECOVERY-When IC Entry removed from DBRC

Problem: There was required to Recovery set databases in an application. Found couple of Databases IC needed for recovery is removed from DBRC. As the Application team have run a NOTIFY.UIC JOB twice wrongly. Even though the GENMAX was set 16, because this problem the required IC was entry was missing.

For example below is one such entry:

NOTIFY.UIC DBD(DBDNAME) DDN(DDNNAME)-
UDATA('RE-INITIALIZE DB, NO IC IS REQUIRED')
DSP0203I COMMAND COMPLETED WITH CONDITION CODE 00
DSP0220I COMMAND COMPLETION TIME 09.088 04:55:00.9

If we issue a NOTIFY.UIC, one entry is getting added to the RECON dataset and oldest entry gets purged. This is the reason for missing IC entry for the required date.

Solution: Inserting/Notifying DBRC about Image Copy

Only if DBRC entry is available IMS will allow recovery. The approach was to remove the oldest entry available and Notify using the following Job, NOTIFY.IC was issued with time stamp from Image copy sysout.

//RCNLST EXEC PGM=DSPURX00
//STEPLIB DD DSN=IMSVS.CA.DBO.RESLIB,DISP=SHR
// DD DSN=IMSSYS.RESLIB,DISP=SHR
// DD DSN=IMSSYS.USER.RESLIB,DISP=SHR
//DFSRESLB DD DSN=IMSSYS.RESLIB,DISP=SHR
//SYSPRINT DD SYSOUT=*
//SYSOUT DD SYSOUT=*
//SYSIN DD *
LIST.DB DBD(DBDNAME) DBDS
DELETE.IC DBD(DBDNAME) DDN(DDNNAME) RECTIME(090822101379)
LIST.DB DBD(DBDNAME) DBDS
NOTIFY.IC DBD(DBDNAME) DDN(DDNNAME) ICDSN(IC.DATASET.NAME) RUNTIME(090801601562) -
FILESEQ(0004) BATCH UNIT(TAPEC) VOLLIST(B00878)
LIST.DB DBD(DBDNAME) DBDS

/*
//

After completion of NOTIFY.IC we ran the Restore job but it failed with :

DFS391I S DBDNAME DDNNAME 090801601562-0400
E N D O F S Y S I N C O N T R O L C A R D S
DFS391I **RECOVER DATA BASE DBDNAME DDNAME DDNNAME
DSP0822I RECOVERY REQUESTED TO TIMESTAMP 09.080 16:01:56.2
DSP0822I DBD=DBDNAME DDN=DDNNAME
DFS2804A HEADER RECORD ON DDNAME DFSUDUMP IS NOT CONSISTENT WITH DBPCB FOR FUNCTION RV
DFS339I FUNCTION RV HAS COMPLETED ABNORMALLY RC=08
************************** Bottom of Data ****************************


To fix this problem Ran IEBGENER to copy the Image copy from TAPE to DASD and HEX ON and found timestamp of the Image Copy. Time stamp was found to be 090801601549. So Delete the existing entry of the image copy and Issued NOTIFY.IC with correct Timestamp.



Below is HEX ON of ImageCopy dataset:
-----------------------------------------------------
.D..DBDNAMEDDNNAME......èpâo._DBDNAME.Y.Y.Y.Y......
0C00CEECCCDECEEDCCDE200020594906CEEDCCDE1E1E1E1E020000
54004321413863274138098F0147261D63274138787878780B0560
-----------------------------------------------------


After making appropriate entry in RECON ran the RESTORE job and recovered the Database.

TIMESTAMP -Calculation Useful Info

TIMESTAMP CONVERSIONS AND EXAMPLES

Assume that local time is PST, offset -08:00
(Offset is always added to UTC to obtain local time.
to obtain UTC from local time, reverse the sign and add.)

An event occuring at:
1998.030 10:00:00.1 -08:00 (punctuated format)
yyyy ddd hh mm ss t
or:
980301000001 -0800 (compressed format)
yydddhhmmsst

is recorded in 12 hex bytes as:
--------------------------------
|1998030F | 18000012 | 3456032D|
--------------------------------
yyyyddd hhmmsstx xxxxfqqs

hh = UTC, 8 hours later than local
x xxxx = low-order time digits ignored by DBRC and you
fqqs = offset:
f = flag bits, normally 0
qq = quarter hours (32 / 4 = 8)
s = sign (D is negative, C is positive)

Given a record identified by a time (such as a PRILOG):

Listed time entered as:
Finds recorded time:
'1998.030 10:00:00.1 -08:00' 1998030F 18000012 3456032D

(Note that DBRC ignores the low-order time digits and the
offset when searching - only the UTC date and time are
significant).

If you supply NO OFFSET, DBRC uses the TIMEZIN value,
which may not be correct.

With the wrong offset:
DBRC looks for the wrong time
'1998.030 16:00:00.1 -02:00' 1998030F 18000012 3456008D

This means that time stamps from different RECON listings,
even with time stamps displayed in different formats, can
be used freely, SO LONG AS THE OFFSETS ARE INCLUDED.

IMS LOG File not closed properly

Problem: IMS IEFRDR Log file did not get closed properly. When we tried using it in the restart for IMSLOGR file, it failed.

Solution: Use DFSULTR0 --- IMS utility to close the LOG file which did not close properly.


Useful info on DFSULTR0 on how IMS log file can be closed.

The following example shows how to close an SLDS created by IMS batch, using DUP mode and ERRC=00000. The input data set is copied to, and closed in, the output data set. The input SLDS information in the DBRC RECON is replaced by the output data set information.
//EXAMPL09 JOB ........
//*
//DFSULTR0 EXEC PGM=DFSULTR0
//* (PARM NOT REQUIRED - SEE NOTES BELOW)
//*
//* NOTE - IMSID= is ignored
//* NOTE - Defaults are DBRC=YES
//*
//SYSPRINT DD SYSOUT=A
//IEFRDER DD ........ Primary SLDS to be closed
//IEFRDER2 DD ........ Secondary SLDS to be closed
//NEWRDER DD ........ Primary output SLDS
//NEWRDER2 DD ........ Secondary output SLDS
//RECONn DD ........ DBRC RECON data set(s)
//* (can be dynamically allocated)
//SYSIN DD *
DUP ERRC=00000
When ERRC=00000 is specified, NEWRDER (and NEWRDER2) is closed when EOF or the first error is encountered on IEFRDER (and IEFRDER2). If the execution is successful, REP mode is not required. If the execution is unsuccessful, DUP mode should be rerun with an ERRC value greater than zero and REP mode is required.


DBRC=YES|NO
Indicates that the DBRC= default is not established by the IMSCTRL macro during IMS system definition.
DBRC=NO (or N) can be specified to explicitly declare that DBRC is not to be used for this execution of this utility.
DBRC=YES (or Y) can be specified to explicitly declare that DBRC is to be used for this execution of this utility. DBRC=YES is required (and the default) for CLS mode. DBRC=YES is optional for DUP and REP modes.
Recommendation: If DUP mode is run with DBRC active, REP mode should also be run with DBRC active


IEFRDER DD
Defines the primary, or only, input SLDS. All input SLDS logs for DUP mode should have the same block size. IEFRDER is used to specify the concatenation of OLDS and SLDS logs for PSB mode. When specifying a concatenation of logs, the names of the logs must be provided in ascending order.
IEFRDER2 DD
Defines the secondary input SLDS. Include this statement only when dual logs are used. Omit this statement if you do not need the data sets. Do not use DD DUMMY or DSNAME=NULLFILE.

NEWRDER DD
Defines the primary, or only, output data set for the new or interim log.
NEWRDER2 DD
Defines the secondary output data set for the new or interim log. If DBRC is active and dual logs are used as input, this statement is required. If DBRC is not active, this statement is not required. Do not use DD DUMMY or DSNAME=NULLFILE.

IMS RECOVER Problem-unrecoverable Indexes Issue

Problem:Backout kept ending with a return code of 0 after application abends, but upon restart of the jobs, it kept saying DUPLICATE records. IMS Backout for Indexes having marked as nonrecoverable in DBRC.

Reason:For databases or indexes marked as UNRECOVERABLE in DBRC, if the abending job is using batch DASD logging, and the archiving process has run BEFORE a backout of any type is submitted, check that the backout job does not use the archived log. This is because when the archive process runs, it will discard any changes to databases marked as UNRECOVERABLE, therefore no changes will be undone by the backout process and the database will not be in a state of integrity related to any other databases in the PSB. The database must then be restored using alternative methods, such as a reload or an index rebuild. The backout will complete with a return code of zero (RC=0) even though no changes were made to the UNRECOVERABLE databases.

Solution:After recovering the primary databse run REBUILD index to bring the database and indexes to a sync point.

Follow up : Requested IBM to update the backout process to notify the user that UNRECOVERABLE databases are participating in a backout using an archived DASD log.

IMS RECOVER

Problem : Received IMS error at REBUILD phase of a IMS Database RECOVER .

“SHARED ACCESS TO VSAM DATA SET DBDNAME NOT ALLOWED BY CATALOG IM13”
“UNABLE TO OPEN DATA SET WITH DDNAME DBDNAME FOR REASON I,33

Resolution:The VSAM definition of the Database file as SHROPT( 3,3). Which is a shared file. So we need to have initial step CHANGE.DB with option NOAUTH before recovery step. And after REBUILD step again we need to add a STEP to Notify DBRC with option CHANGE.DB DBD(DBDNAME) AUTH

Additional Information:
CHANGE.DB DBD(DBDNAME) NOAUTH will mark the Database as no share in the DBRC.

CHANGE.DB DBD(DBDNAME) AUTH will again mark the database as sharable in DBRC.

Always when dealing with VSAM files with SHR (3,3) this need to be taken care.

IMS GSAM File -Space Probelm

Problem: Job abend due to space problem in the GSAM File. GSAM File had used only TRK(1,1), even though the allocation for the GSAM File was TRK(470, 470).

Reason: Space archive job runs to free unused space in the sytem had run before the abended job was restarted.

Solution:Create a new version of GSAM GDG file by coping the old GSAM file data with RECFM=U (This is done so the GSAM is copied as it is, no formatting is done) and in the next step the RECFM is changed to FB. We need the overrides the GSAM file reference to use the new file or you can rename new the GSAM File to the old file name and use it.

IMS Log Analizer-To get changes happened to IMS Database

How IMS Log can be read for updates done to IMS Database

Solution
Below control card can be used to get all updates done to the database DBNAME . Which extracts database records ( X’50’ ) with all Subtypes for a given database.

//SYSIN DD *
CONTROL CNTL DDNAME=LOGIN,DDNOUT=LOGOUT
OPTION COPY PRTSYS=Y,OFFSET=5,FLDTYP=X,VALUE=50,FLDLEN=1,COND=M
OPTION COPY PRTSYS=Y,OFFSET=53,FLDTYP=C,VALUE=DBNAME,FLDLEN=8, X
COND=E
END

Just for your information , we used ILOGREC for getting IMS Log record structure।

*********************************************************************** ** D A T A B A S E X ' 5 0 ' L O G R E C O R D ** ** ** ** SUBCODE X'50' RECORDS CONTAIN UNDO DATA FOR DATA BASE ** ** BACKOUT AND REDO DATA FOR DATA BASE RECOVERY. ** ** SUBCODE X'51' RECORDS HAVE NO DATA AND INDICATE THE ACTION ** ** LOGGED BY THE PRIOR RECORD WAS UNSUCESSFUL. ** ** SUBCODE X'52' RECORDS CONTAIN UNDO DATA LOGGED BEFORE ** ** AN INSERT TO A KSDS DATASET. SUBSEQUENT SUBCODE 50/51 ** ** RECORDS INDICATE WHETHER THE INSERT WAS SUCCESSFUL. ** ***********************************************************************
Sample JCL



//STEPM EXEC PGM=DFSERA10,REGION=6M

//STEPLIB DD DSN=IMSVS.IMSSYS.RESLIB,DISP=SHR

//SYSPRINT DD SYSOUT=*

//LOGIN DD DSN=READ.LOGFILE,UNIT=TAPEC,

// DISP=(OLD,KEEP,KEEP),VOL=SER=F07113

//LOGOUT DD SYSOUT=*

//SYSIN DD *

CONTROL CNTL DDNAME=LOGIN,DDNOUT=LOGOUT OPTION COPY PRTSYS=Y,OFFSET=5,FLDTYP=X,VALUE=50,FLDLEN=1,COND=M

OPTION COPY PRTSYS=Y,OFFSET=53,FLDTYP=C,VALUE=DBNAME,FLDLEN=8, X

COND=E END