Reprinted with permission. ©1992 Candle Corp. All Rights Reserved.
In today's world of changing platforms, keeping track of the exact parameters that went into the initialization of an MVS operating system (IPL) can be an onerous task. With PR/SM becoming standard on ES/9000 processors, sysplex definitions, and Advanced Program to Program Communications (APPC), keeping track of what is happening becomes more important and more difficult. This article will address full utilization of the base MVS/ESA features that are available to you for documenting your system as you IPL and as you modify it. Self documentation helps you to clearly identify the system components and parameters that were in effect when a problem or question arose.
Why should anyone build a self-documenting system? There are several reasons: In a test environment, the more the system documents, the less test time and effort required by the programmer; when diagnosing historic production system problems after they occurred (not as they are happening), the more documentation the system collected, the less research required to reconstruct that environment. Event when trouble-shooting a system problem as it is occurring, system-generated documentation guarantees accurate records of what parameters were in effect, not just what an operator or systems programmer meant to place in effect. You already use some of these options; just expand you use of the facilities in order to reduce your time spent on documentation efforts.
There are several levels of support and effort required to make you system self-documenting:
Each of these levels provides totally different types of information and requires different amounts of effort. The four levels derive support from two system services: SYSLOG and System Management Facility, or SMF. Each offers a distinct perspective on the events recorded. In most cases, there is little overlap in what they report, that is, events reported on SYSLOG are generally not written to SMF.
SYSLOG based documentation provides a broad-brush, chronological, system-wide trail. SYSLOG details what was happening throughout the system when the environment was defined or changed. For example, based upon the message flow, it is apparent that an event occurred while production jobs were running. This makes SYSLOG ideal for viewing the sequence of events of interest.
SMF based documentation provides event recording at a finely detailed level, without regard for any interconnection of events. This recording emphasizes what was done, by whom, and when it happened. For example: "User XYZ updated file THIS.FILE at 20.00 on 12/11/92." It is well suited for analysis and comparison of the discrete specifics of the environment.
In order to fully appreciate what has evolved in MVS, a brief look at its history is appropriate. Prior to MVS/XA 2.2, MVS was a very static operating system. It took a system generation (SYSGEN) to change most system parameters. In MVS/XA 2.2, many of the SYSGEN parameters and options were moved into SYS1.PARMLIB. However, the parameters were still pretty static; changes were only available at IPL. MVS/ESA 3.1 and subsequent releases provide dynamic change support via the MODIFY and SET operator commands. Many more of the system tasks and services are adding informational display support with the operator DISPLAY command or a MODIFY command requesting status. Further details on the history of SYS1.PARMLIB can be found in the Candle Computer Report, June 1992.
MVS started out as an isolated operating system. It "grudgingly" shared some resources with other MVS systems (or other systems). First there was the hardware RESERVE to prevent multiple updates to the same DASD dataset. RESERVEs carried a hefty penalty on affected system's performance: No other system could access the DASD volume while it was RESERVEd by another system. Global Resource Sharing (or similar products) connected multiple systems with "friendlier" sharing. GRS notified the other systems of DASD update activity and they in turn would not impact the DASD resource being updated. With MVS/ESA 4.x we have multiple MVS systems composing a systems complex (sysplex). A sysplex shares consoles and an External Time Reference (ETR), in addition to DASD. In a PR/SM environment, one MVS system can take over the resources of a failing logical partition (LPAR) and pick up its workload! In addition, the formerly cloistered systems are now sharing parameter sets (such as console definitions), computer operators and more.
All of this provides for terrific reliability, availability and service. However, it makes keeping track of the specific environment in which jobs are running very difficult. It also makes auditing the system quite a challenge.
MVS has grown more cooperative as it has evolved. Every time MVS IPLs, a standard collection of messages is produced (exactly which messages are logged varies by release). These messages are logged automatically every IPL.
System Parameters - Not only has MVS been learning how to interact with other MVS system images, it has been changing within the confines of a single system. MVS/ESA 4.x introduced iplparm which shows you where to find initial IPL parameters. IPL parameters now come from (potentially) two different libraries. IPL parameter collection is two-phased: phase one determines the very basic parameters for the IPL and is directed by device address; phase two determines the higher-level parameters for the system and is directed by the master catalog.
Phase One - The operator specifies the traditional Load Unit Address (sysres or IPL volume) and a new Load Parm which includes the iplparm device address. Initial Program Load (IPL) searches the iplparm volume for SYSn.IPLPARM or SYS1.PARMLIB for members LOADxx (as of 4.1) and NUCLSTxx (as of 4.2.2). After processing those members, iplparm is closed. SYSLOG records which dataset provided these parameters. Please note that "MVS/ESA SP Version 4 Implementation Guide, GG24-3628-01" recommends using SYSn.IPLPARM data sets, instead of SYS1.PARMLIB to store LOADxx and NUCLSTxx members.
Phase Two - When IPL is ready for the remaining parameter sets, the master catalog (defined by phase one) locates which SYS1.PARMLIB (also referred to as parmlib) to use. This is significant because MVS/ESA 4.1 allows SYS1.PARMLIB to be on a removable volume which can be unloaded after IPL completes. If parmlib is later required by system processes, such as a SET command, a mount request will be issued for that volume.
Nucleus Construction - MVS/ESA 4.1.1 restructured the nucleus. It is now customized at IPL instead of loaded as a single module from SYS1.NUCLEUS. Using NUCLSTxx, a systems programmer can add installation modules to the nucleus (e.g., user SVCs) and replace nucleus-resident modules (by delete and insert pairs of operations). SYSLOG reports on module deletions, but not includes, during nucleus build (see Figure 1).
System Parameter Set Specification - IEASYS00 was always used in pre-MVS/ESA 4.2 systems. The operator could only specify an additional IEASYSxx member (SYSP=xx). This enabled systems programming to lock selected parameters regardless of alternate IEASYSxx members. In MVS/ESA 4.2, the new LOADxx tells IPL which IEASYSxx member to use, exclusive of IEASYS00, if desired. This increases the importance of a SYSLOG review to determine which IEASYSxx member(s) were used. See Figure 1.
IEA371I SYS1.IPLPARM ON DEVICE 0143 SELECTED FOR IPL PARAMETERS
IEA246I LOAD ID 00 SELECTED
IEA519I IODF DSN = SYS1.IODF00
IEA520I NUCLEUS 1 SELECTED
IEA370I MASTER CATALOG SELECTED IS ICFCAT.VMVS422
IEA372I DUMBR14 EXCLUDED FROM NUCLEUS
*IEA247I USING IEASYS00 FOR RELEASE 3.8 , VERSION SP4.2.2 JBB4422
. . . .
IEA191I CONSOLE 660 DEFINED AS MASTER
IEA168I VATLST00: VATLST DEFAULT USE ATTRIBUTE OF PRIVATE USED.
=====================================================================
IEA713I LPALST CONCATENATION LPA=(xx,L)
SYS1.LPALIB
SYS1.ISPF.V3R3.LPALIB
. . . .
IEA331I LINK LIBRARY CONCATENATION LNK=(xx,L)
SYS1.LINKLIB
SYS1.MIGLIB
IEE949I 06.13.53 SMF DATA SETS ### LISTDSN in SMFPRMxx
NAME VOLSER SIZE(BLKS) %FULL STATUS
P-SYS1.MANx MVS422 900 92 ACTIVE
S-SYS1.MANy MVS422 900 100 DUMP REQUIRED
Figure 1 - SYSLOG listing Level I and Level II output
By updating the IEASYSxx director values for select parameter sets, several SYS1.PARMLIB members will log their contents at IPL. Exactly which members provide the logging depends upon the level of MVS in use. The logging occurs when their IEASYSxx specifications include the ",L" operand, or if and when an appropriate parameter is set within the member. While less than one half of the SYS1.PARMLIB members currently provide this facility, using those that do will provide invaluable documentation , as shown in Figure 1.
The parmlib directors which support documentation is IEASYSxx are:
CLOCK=(xx,L) FIX=(xx,L) IPS=(xx,L) LPA=(xx,L) OPT=(xx,L) SCH=(xx,L) CON=(xx,L) ICS=(xx,L) LNK=(xx,L) MLPA=(xx,L) PAK=(xx,L) SVC=(xx,L)
Two members of SYS1.PARMLIB contain parameters to expand upon the SYSLOG documentation listing: IEFSSNxx where PROMPT=DISPLAY can be specified on the IGDSMS entry, and SMFPRMxx in which LISTDSN can be specified.
There are multiple operator commands which provide information about the status and controlling parameters in your system. These can be issued by an operator, through a production JES job stream (depending upon your "operator commands in JCL" parameter setting), or via AF/OPERATOR®. SYSLOG captures the operator commands.
Following are the most common operator commands:
Command Information parmlib
D APPC Advanced Program to Program APPCPMxx
S APPC,...,L Communication: display current
T APPC=(xx,L) cumulative results, start listing
members, change (add to) existing
environment.
D ASCH APPC Scheduling: display current ASCHPMxx
S ASCH,...,L cumulative results, start listing
T ASCH=(xx,L) members, change (add to) existing
environment.
D CNGRP Console Groups (used for alternate CNGRPxx
T CNGRP=(xx,L) console processing)
D CONSOLES Display console configuration CONSOLxx
F DLF,STATUS Display Hiperbatch parameters COFDLFxx
D ETR,DATA Display External Timer Reference CLOCKxx
D IOS,CONFIG I/O Configuration information N/A
D MMS Display MVS Messaging Service MMSLSTxx
D MPF Display Message Processing Facility - MPFLSTxx
especially .CMD and USEREXIT entries
D SMF,S System Management Facility data sets SMFPRMxx
D SMF,O (,S) or current options (,O) -
include SETSMF command settings
D SMS,A Storage Management Subsystem defaults IGDSMSxx
D SMS,OPTIONS (,A) or current options (,OPTIONS) -
include SETSMS command settings
D XCF,SYSPLEX System complex status display - also COUPLExx
D XCF,COUPLE results of SETXCF commands XCFPOLxx
D XCF,PRSMPOLICY
T SCH=(xx,L) Change and list the Program SCHEDxx
Properties Table.
Note: there is no D SCH command
Figure 2 - Operator Commands Table
SET TIME (01) SET DATE (02) SET IPS (04) SET SMF(05) SWITCH SMF (06) HALT EOD(07) IPL prompt (08) IPL SMF (09) IPL SRM (10) SET OPT (11) SET ICS(12) SETSMF(13) SET MPF (14) Restart SMF(15) SET DAE (16) SET PFK(17) SET GRSRNL(18) SET APPC(19) SET ASCH (20) SET SCH (21)
Figure 3 - SMF Record 90 Subtype values -- Italicized entries are important to system security.
The fourth and final level of self-documentation requires that the user build JCL and procedures which will manually collect the desired information, store the results for later review, and format SMF records into meaningful reports. Depending upon the amount of information desired, these can be simple tasks or significant efforts.
Several MVS components write valuable SMF records. However, there is no native MVS formatting utility to use the captured data. The installation programmer must decide what information to extract for reporting. The specific records, their subtypes (if any) and exact contents will vary depending upon the level of MVS you are running. Review the Systems Management Facility manual for your specific environment. In all cases, system documentation for critical records 00 (IPL), 07 (SMF data lost) and 90 (system environment) are available, but not all subtypes of SMF90 will be produced. Also note the following:
Currently, there are 21 events journalled by the SMF90 record. In Figure 3, the italicized entries are important to system security. When you format a report using these records, several patterns will become apparent:
This analysis also detects critical operations oversights. A HALT EOD (Z EOD) command should always be issued before the system is QUIESCEd. HALT EOD performs many valuable functions: updating SYS1.LOGREC, closing out SMF buffers, and other services. The system should not be quiesced or rest until a Z EOD is at least attempted. Watch for a 90-07 record (HALT EOD) before an IPL record (SMF00) or IPL SMF (90-09) to ensure this procedure is being followed.
In order to facilitate your use of these records for proper reporting, using IFASMFDP (or other SMF data pre-processor type program) is highly recommended. This creates a separate, daily SMF data set containing on the records you desire.
AF/OPERATOR can further aid you in capturing system environment changes. Not every SET command (and no MODIFY command) writes an SMF90 record. AF/OPERATOR can intercept SET commands and write a record of its own to a log dataset. The resulting log can be used to augment the SMF90 report shown in Figure 4. The command modification exited provided in MVS/ESA 4.1 provides an exit point for the systems programmer to perform a similar function , event wring an installation version of the SMF90 record.
System Status SMF Record Analysis SMF Records 90 and User Records for SYSID: SYS9 for October 17, 1992
Date and Time Revu Action New Val Old Val Other 10/17 @ 13:02 IPL SMF 10/17 @ 13:02 IPL Tuning IEAICS00 IEAIPS00 IEAOPT00 10/17 @ 13:03 SET DAE ADYSET00 10/17 @ 13:03 SET MPF MPFLST09 10/17 @ 13:15 SET APPC APPCPM00 10/17 @ 13:15 SET APPC APCPMMH 10/17 @ 14:02 MODIFY DLF COFDLF99 10/17 @ 15:25 SET MMS MMSLST90 10/17 @ 17:40 XX SET SCH SCHED09 10/17 @ 17:56 XX SWITCH SMF SYS1.MAN3 10/17 @ 18:22 HALT EOD 10/17 @ 18:57 IPL SMF
Notes: 90-06 normal SWITCH SMF records are suppressed. Italicized events are
from user records (SMF or other)
Figure 4. Sample SMF90 Report
Another key role that AF/OPERATOR and the MPF command modification exit can provide is correcting "incomplete" SET commands issued by the operator or systems programmer. For example, the person issuing the command forgot to request the ",L" list option of one of the SET command s which will accept it. The exit can intercept and reformat the command. This can be especially valuable in system tuning efforts. By forcing the ",L" operand on SET ICS, IPS and OPT commands, a complete set of the new parameters is logged to the console. This serves to document exactly what was set and when the change occurred. The value of this option is apparent when converting to the completely rewritten tuning members (IEACSxx, IEAIPSxx and IEAOPTxx) of MVS/ESA 4.2.
The command modification exit is specified in the MPFLSTxx member of parmlib as .CMD USEREXIT(nnn). Also, when the command modification exit changes the text of an operator command, both the input and modified command strings are written to SYSLOG. This ensures that the actions of the exit are documented.
Another option collects a series of DISPLAY and MODIFY commands and has them issued on a periodic basis, such as once a day. The output from these commands will be listed on SYSLOG. AF/OPERATOR can also collect the command responses into a separate log file for easier review. This will provide a snapshot of the environment at that time, enabling you to more easily reconstruct the environment.
Scheduling these commands can be accomplished by JES or AF/OPERATOR. Both can issue commands based on time of day or interval (e.g., every six hours). Or you can trigger differing sets of displays based upon console messages (using AF/OPERATOR or the MVS Message Processing Facility (MPF)). For example, automatically issuing a MODIFY DLF,STATUS command when a console message acknowledges a MODIFY DLF,NN=nn command completed.
Batch jobs are a key part of system documentation. Each IPL could submit a batch job using AMBLIST to save a nucleus and Link Pack Area map. This listing could then be used to determine exactly when a change was made to a critical system module.
Don't overlook the security system as a source of system parameter documentation. Changes made to either global system options or strategic resource profiles are very important. Formatting these reports frequently requires a batch job.
Set up a requirement that system utility packages document their changes to the operating system environment. For example, OMEGAMON® will optionally write a SYSLOG message, SMF record, or both for specified commands. Commands like APFUA and APFUD (add or delete an APF library list entry), or LPAM (modify the Link Pack Area) should be self-documenting.
DELTAMON® for MVS documents changes to the system environment (some MVS and JES2 parameters, hardware configuration, and data set contents). DELTAMON produces logging output detailing whatever was found in the systems' internal control blocks at a specified time, such as the APF-library list. These reports supplement our system documentation and may also alert you to undocumented changes made to your system.
OMEGAMON can also be used to document the exact parameters settings at a specific time. With its time screens facility (TSF) and screen logging facility, it is an easy operation to create a set of system documenting screens which automatically execute and log at any desired time or interval.
The MVS operating system is evolving rapidly - as are the environments it must support. As evolution progresses, IBM is providing more documentation for review and analysis. It is our job to use that information to document our environment, whether to demonstrate improvements brought about by tuning parameter changes, provide documentation detailing our environment for problem determination, or simply to answer specific questions about our system environment at a given time.
How much effort do you want to put into reconstructing system documentation? Some of the information is already provided (Level I); more information can be collected after a one time change to parmlib (Level II); your operator (manual or automated) can request system status information (Level III); or you can ambitiously collect system environment information (Level IV). These are not mutually exclusive, nor are they required. The information is there for your selection.
Consider carefully: If the audit trail is WTO based, SYSLOG will store critical information about changes to your system. If the audit trail is SMF based, either the product or your own report generation software can be used to format the audit trail. Of course, having both offers the greatest flexibility.
Remember the more documentation the system collects for you, the less you have to write.
Please route any comments or feedback to:
mhahnbe@pobox.com