http://www.euy2k.com/audit1.htm To: Multiple recipients of list Subject: Monticello Y2K Audit Report From: John Voglewede Date: Mon, 5 Oct 1998 17:41:43 -0400 (EDT) A text copy of NRC's Audit Report on Monticello Nuclear Generating Plant's Year 2000 Program is provided below. This is the first plant subject to NRC's Y2K audit program. An HTML version of this document will be posted on NRC's Web site very shortly. - - - - - - - - - - - - - - - - - - - - - October 2, 1998 Mr. Roger O. Anderson, Director Nuclear Energy Engineering Northern States Power Company 414 Nicollet Mall Minneapolis, Minnesota 55401 SUBJECT: MONTICELLO NUCLEAR GENERATING PLANT - AUDIT REPORT ON THE YEAR 2000 PROGRAM (TAC NO. MA1858)

Dear Mr. Anderson:

On September 15-17, 1998, the NRC staff conducted an audit of the Year 2000 (Y2K) program at the Monticello Nuclear Generating Plant as a followup to NRC Generic Letter (GL) 98-01, "Year 2000 Readiness of Computer Systems at Nuclear Power Plants," issued on May 11, 1998. The enclosed report presents the results of the audit. The results of this audit and subsequent audits at other selected nuclear power plants will be used by the staff to determine the need for additional action, if any, on Y2K readiness for nuclear power plants.

In accordance with 10 CFR 2.790 of the NRC's "Rules of Practice," a copy of this letter and its enclosure will be placed in the NRC Public Document Room. Please contact me at (301) 415-1392 if you have any questions or comments on the enclosed audit report. 

Sincerely,

Original Signed by  
                         
Tae Kim, Senior Project Manager
Project Directorate III-1 
Division of Reactor Projects - III/IV
Office of Nuclear Reactor Regulation

Docket No. 50-263

Enclosure:  As stated

- - - - - - - - - - - - - - - - - - - -
U.S. NUCLEAR REGULATORY COMMISSION
OFFICE OF NUCLEAR REACTOR REGULATION (NRR)
               AUDIT REPORT 
                   ON
 IMPLEMENTATION OF GENERIC LETTER (GL) 98-01
 "YEAR 2000 READINESS OF COMPUTER SYSTEMS AT
           NUCLEAR POWER PLANTS"


Docket No:50-263
License No: DPR-22
Licensee: Northern States Power Company (NSP)
Facility:Monticello Nuclear Generating Plant (MNGP)
Location:2807 West Highway 75
         Monticello, MN 55362
Dates:   September 15 - 17, 1998
Audit Team Members: Matthew Chiramal, NRR
                    Dave Butler, Region III
                    Deirdre Spaulding, NRR
Approved by:    Jared Wermiel, Chief
              Instrumentation and Controls Branch
              Office of Nuclear Reactor Regulation

EXECUTIVE SUMMARY

From September 15 through 17, 1998, the NRC staff conducted an audit of the Year 2000 (Y2K) program at the Monticello Nuclear Generating Plant in accordance with the audit plan (Attachment 1) for this activity. The purpose of the audit was to (1) assess the effectiveness of the Northern States Power Company (the licensee) programs for achieving Y2K readiness, including continued safe operation of the plant as well as compliance with applicable NRC regulations and license conditions with respect to the potential Y2K problems, (2) evaluate Y2K program implementation to assure that the licensee's schedule is in accordance with NRC Generic Letter (GL) 98-01 guidelines for achieving Y2K readiness by July 1999, and (3) assess the licensee's contingency plans for addressing risks associated with potential events resulting from Y2K problems. The audit team reviewed selected licensee documentation regarding Monticello's Y2K program and conducted interviews with the cognizant licensee personnel. The results of this audit and subsequent audits at other selected plants will be used by the staff to determine the need for additional action, if any, on Y2K readiness for nuclear power plants.

Based on the staff's assessment and evaluation of the Monticello Y2K readiness program, the following observations were made:

  1. The Monticello Y2K readiness program is comprehensive and is based on the nuclear power industry Y2K problem guidance contained in Nuclear Energy Institute (NEI)/Nuclear Utilities Software Management Group (NUSMG) 97-07, "Nuclear Utility Year 2000 Readiness."
  2. The Monticello Y2K readiness program is receiving appropriate management support and oversight.
  3. The licensee began the formal Monticello Y2K program later than most licensees (June 1998) and as a result, the licensee is still in the initial assessment stage. The licensee is undertaking an ambitious schedule in order to meet the July 1999 Y2K readiness date established by the NRC staff in GL 98-01. Despite the late start, the Y2K readiness schedule appears to be achievable because of the limited number of software items at the site, the fact that the licensee has already begun remediation of major critical computer systems, and the licensee has received support via information sharing with the Boiling Water Reactor Owners Group and a utility alliance.
  4. The licensee has not started the Monticello Y2K contingency planning. The licensee plans to utilize the nuclear industry guidance in NEI/NUSMG 98-07, "Nuclear Utility Year 2000 Readiness Contingency Planning," for this effort. With proper attention provided by management, the licensee should be able to complete this effort by July 1999.
  5. The licensee's corporate and Monticello plant-specific Y2K program interfaces are effectively addressing grid reliability and availability issues.
  6. The licensee will address the operating status of Monticello, which is currently planned to be in a refueling outage on December 31, 1999, in its corporate Y2K readiness plan and associated contingency planning. Both operating and shutdown conditions for Monticello will be considered. REPORT DETAILS

1.0 INTRODUCTION

The objectives of the Monticello Nuclear Generating Plant (MNGP) Y2K Program audit were to:

  1. assess the effectiveness of the Northern States Power Company (the licensee) programs for achieving Y2K readiness including continued safe operation of the plant as well as compliance with applicable NRC regulations and license conditions with respect to the potential Y2K problems,
  2. evaluate Y2K program implementation to assure that the licensee's schedule is in accordance with NRC Generic Letter (GL) 98-01 guidelines for achieving Y2K readiness by July 1999, and
  3. assess the licensee's contingency plans for addressing risks associated with potential events resulting from Y2K problems.

The audit was conducted in accordance with the established audit plan (Attachment 1) which was based in part on the guidance and requirements contained in the following documents:

  • GL 98-01, "Year 2000 Readiness of Computer Systems at Nuclear Power Plants"
  • Licensee Response(s) to GL 98-01
  • Plant technical specifications and license terms and conditions
  • Applicable NRC regulations
  • NEI/NUSMG 97-07, "Nuclear Utility Year 2000 Readiness"

Prior to the audit at the plant site, the audit team had obtained and reviewed the MNGP Year 2000 Readiness Implementation Plan and associated work instructions (draft versions of document numbers 1, 2, and 3 listed in Attachment 2).

The audit process started with an entrance meeting attended by the MNGP Year 2000 Readiness Project Manager (PM) and other site personnel, the Year 2000 PM of Prairie Island Nuclear Plant, and members of the audit team. Attachment 3 is a list of the attendees. The PM and members of the project team described the project organization, the project plan and its implementation, and the project status and ongoing activities.

Following the meeting, the audit team spent the rest of the audit reviewing the project plan and its associated procedures, the plan implementation products (documents and data bases) and interacting with the project team members, particularly with the PM. The documents reviewed are listed in Attachment 2.

2.0 MNGP Y2K PROJECT DESCRIPTION

2.1 Project Organization

The MNGP Y2K project has 15 full-time persons (including the PM and two contractors) and 3 part-time persons). The PM has overall responsibility for the project and reports to the General Superintendent - Design and Engineering, MNGP, who reports to the Monticello Site Plant Manager. The Plant Manager reports to the President-Nuclear Generations who provides the information to the Project Sponsor.

MNGP participates with other organizations that are addressing the Y2K effort. The licensee has been involved with the Boiling Water Reactor (BWR) Owners Group. According to the licensee, the BWR Owners Group final report on its Y2K program is due October 1998. MNGP has been able to take advantage of the BWR Owners Group generic Y2K efforts for several noncompliant computer systems. MNGP will be upgrading its nonsafety-related process computer system (PCS), a General Electric Company (GE) 3D Monicore Baseline 94 system. The GE 3D Monicore Baseline 98 upgrade system which is Y2K compliant is scheduled to be installed at the site in November 1998 with testing and final acceptance of the PCS occurring over a period of approximately 2 months. Other Y2K compliant upgrades being coordinated through the BWR Owners Group are the GE NUMAC automated TIP [traversing in-core probe] control units and the rod worth minimizer (RWM), and GE Fanuc reactor recirculation control and motor-generator (MG) set scoop tube positioner and controller systems.

To further the exchange of Y2K information, MNGP is also part of a Y2K Alliance, which is composed of representatives from Point Beach, Kewaunee, Monticello, Duane Arnold, and Prairie Island nuclear power plants.

2.2 Project Plan

The MNGP Year 2000 Readiness Implementation Plan (Item 1 of documents reviewed in Attachment 2) is the plant-specific plan that was developed by the licensee and issued on July 17, 1998. It is based on the guidance provided in NEI/NUSMG 97-07, which was endorsed by the NRC in NRC GL 98-01 as guidance that when properly implemented presents one approach for achieving Y2K readiness. The audit team's review confirmed that the MNGP Year 2000 Implementation Plan is based on the guidance contained in NEI/NUSMG 97-07.

The MNGP Year 2000 Readiness Implementation Plan consists of the following phases: awareness, initial assessment, detailed assessment, remediation, contingency planning and risk management, and notification. It also includes requirements for quality assurance, regulatory considerations, and documentation.

2.2.1 Awareness

At MNGP, the formal Y2K awareness phase of the Y2K program was initiated in June 1998 to all site personnel via Site News Letters. Additionally, through various group meetings and e-mails, information on Y2K problems was disseminated to system engineers and staff during the start of the project. On September 16, 1998, the plant Year 2000 Project was discussed at the Engineering/Technical Staff Training session.

At the corporate level, Y2K awareness began in 1996. The NSP board of directors approved the NSP Year 2000 Project and its budget in 1996.

The MNGP Y2K project implementation schedule is provided in Table 1.

2.2.2 Initial Assessment

The initial assessment stage of the MNGP Y2K Project started in July 1998. The completed initial assessment will result in the identification of all software applications and embedded system components at the MNGP. The NSP Software Master Configuration Index (SMCI) was used by MNGP to identify the software applications, and the Champs database was used to aid in the identification of embedded systems. The tasks of initial assessment include (1) inventory, (2) categorization, (3) classification, (4) prioritization, and (5) analysis of the initial assessment. The licensee indicated that the inventory of all software application items and approximately 80 percent of the embedded system components was complete.

In the identification of embedded systems, it is necessary to review the procedures and documentation for occurrences of phrases that would indicate the existence of an internal clock or processor, survey the vendors for information on their equipment, perform system walk-downs, and review schematics, program listings, and reference manuals.

Table 2 provides the results of inventory of software items. Of the 290 software items identified, 120 will require assessment testing, and 60 will require additional detailed and integrated testing. Table 3 provides a list of safety-related softwares at MNGP.

Table 4 provides the results of inventory for the embedded systems. A total of 453 embedded items had been identified to date. Out of the total of 453 identified embedded items, 175 still need to be assigned a classification

Prioritization

The inventory phase includes the prioritization of the identified items. The priority is based on the criticality and risk of the functions performed. The criticality is based on the criteria as suggested by NEI/NUSMG 97-07: (1) critical (life-threatening implications, required by regulations; major impact on service to customers), (2) severe (mandated by regulatory agencies but can be lost for short periods of time; asset is used solely as a backup to an asset of critical importance; business continues but with great difficulty), (3) high (mandated by regulatory agencies but which have compensatory measures; business continues but with serious difficulty), (4) medium (minimal impact on company's core business; compensatory measures are more costly to use than the asset), (5) low (customer service is not affected; minimal impact on business operation), or (6) none ( no lost productivity; asset is no longer being used or has no identified users). Risk assessment is based on the frequency of usage and type of usage and is classified as critical, high, medium, or low. Priority of high, medium, or low will be assigned commensurate with the level of importance relative to criticality and risk.

Analysis of Initial Assessment

The results of the MNGP initial assessment of the software applications and embedded items will be placed in the MNGP Y2K Application Checklist and Embedded Component Summary.

Analysis of the initial assessment is the final step in the initial assessment phase. During the analysis of the initial assessment, items are dispositioned as "not affected" or designated as needing further detailed assessment. Items that do not display a date or calculate a date require no further evaluation and are designated as "not affected." All other items will require detailed assessment and will be dispositioned as follows: use as is, remove, replace, or remediate and test.

NRC Audit Team Assessment

Several folders for embedded components were reviewed by the audit team. The components were selected from the database print-out titled "Embedded Components Sorted by Classification [sp]" dated Tuesday, September 15, 1998, consisting of 18 pages.

Out of a total of 453 embedded components identified, a total of 32 embedded component folders were reviewed by the audit team.

159 items had a classification that needed to be determined - 24 items were selected for review

23 items were classified under "Continuity of Business" - 1 item was selected for review

126 items were classified under "Important to Operation" - 2 items were selected for review

12 items were classified under "License Commitment" - 1 item was selected for review

55 items were classified under "Non-essential" - 0 items were selected for review

12 items were classified under "Personnel Safety" - 1 item was selected for review

20 items were classified under "Required by Regulations" - 1 item was selected for review

8 items were classified under "Safety Related" - 2 items were selected for review

While reviewing the embedded component information, the audit team found that for component with ID number 427 a "low" priority was assigned. The PM indicated that a Y2K issue does not exist here because there is no date function. From its initial look at the folder in detail, the audit team determined that there seemed to be a different method for determining the priority of these components from the method that is spelled out in the NEI/NUSMG 97-07 guidance. The PM indicated that the impact evaluation grid, risk evaluation grid, and corrective action grid proposed in NEI/NUSMG 97-07 were modified and combined in the risk assessment and prioritization guidance provided in the MNGP Year 2000 Embedded Component Work Instruction (item 3 of documents reviewed in Attachment 2). The intent was to make risk assessment and priority determination easier. The PM also indicated that the determination of risk and priority also involves the engineering judgements of the evaluator, system engineer, system superintendent, and PM. The audit team considered the explanation acceptable and for components in which no date functions exist a low priority is appropriate. Table 5 provides a list of embedded components that were reviewed by the audit team. Table 6 provides information on the embedded components that MNGP classified as safety related.

2.2.3. Detailed Assessment

In the detailed assessment phase, MNGP will obtain information on each item to determine its expected performance when subjected to the NEI/NUSMG 97-07 identified problem dates. There are four different evaluations that may be carried out during the detailed assessment phase. Vendor evaluation, plant-owned or supported software evaluation, interface evaluation, and embedded components evaluation. Vendor evaluation encompasses validation testing based on the criticality of the item, prior experience with the vendor, extent of documentation, or plant knowledge of the item. Plant-owned or supported software evaluation encompasses knowledge-based decisions, scanning, and testing. When testing is proposed, test specifications and procedures are developed. Interface evaluation encompasses the review of the interface capability with software and applications that interface with other systems. Embedded components evaluation encompasses the use of knowledge-based decisions and testing. When sufficient vendor and plant information is available to support a knowledge-based decision, no additional testing is required. Upon completion of the detailed assessment, each component found to be susceptible to the Y2K problem will be used as is, retired, replaced, or modified.

2.2.4. Y2K Testing and Validation

MNGP will perform Y2K testing in support of the evaluation efforts to determine whether the Y2K problem is present. Testing is performed during detailed assessments and requires the development of test procedures. Y2K testing will also be performed subsequent to remediation to determine whether those efforts have eliminated the Y2K problem and no unintended functions are introduced.

MNGP will perform assessment testing per computer problem/change reports (PCRs) and associated verification and validation (V&V) plans and test procedures that they currently have or will establish. Assessment testing will be handled as follows: The test procedures will be written as the application or process software is received and evaluated. A generic test procedure has been prepared which is being used as the starting point. It consists of 16 various categories for Y2K evaluation and testing. Some test procedures, such as those for the security computer and equipment database, are currently being developed from the generic test procedure. This assessment testing process is expected to continue through January 1999.

MNGP will perform testing subsequent to remediation consisting of unit testing, integration testing, and system testing. Unit testing focuses on a single application, software module, or component. Integration testing examines the integration of related software modules, applications, and components. System testing examines the hardware and software components of the system as a whole.

MNGP will perform validation to confirm that the software is capable of performing its intended function. Validation is performed subsequent to remediation and Y2K testing. Upon satisfactory validation, certification and documentation will indicate "Y2K Ready" or "Y2K Compliant" depending on the remediation that was implemented.

2.2.5. Remediation or Replacement

Remediation or replacement will be performed per PCRs and associated V&V plans. A review of the SMCI for final disposition will also be performed. The purpose of remediation is to properly disposition items identified in the detailed assessment. MNGP is revising its existing "Computer & Information Systems - Problem/Change Report" (item 6 of documents reviewed) for software applications, and "Condition Report Process," (item 5 of documents reviewed) for embedded systems. These two documents ensure that identified items are properly tracked and dispositioned.

2.2.6. Regulatory Considerations

The MNGP Year 2000 Readiness Implementation Plan and associated documents (items 1, 2, 3, and 4 of documents reviewed) include references to existing plant procedures that have guidance on regulatory considerations, such as 10 CFR 50.59 reviews, and reportability evaluations per 10 CFR 50.72, 10 CFR 50.73, and 10 CFR Part 21, and operability determinations as required by plant technical specifications.

2.2.7. Contingency Planning

MNGP has not begun contingency planning; however, in January 1999 MNGP will begin its contingency planning in accordance with NEI/NUSMG 98-07, "Nuclear Utility Year 2000 Readiness Contingency Planning."

2.2.8. Y2K Program Management

With regard to the MNGP schedule, there are activities that need to be completed by individuals at the NSP corporate level beyond the control of the MNGP Y2K team. Thus, when making the determination whether the MNGP Y2K project is on schedule, the audit team evaluated the interaction of the MNGP Y2K project management with the NSP Y2K corporate Y2K program.

2.2.9. Electric Grid Issues

MNGP is addressing the issue of substation equipment in the following manner. There appeared to be some questions as to where the boundaries of responsibility for review of substation equipment reside. The boundary between NSP generation and the new independent transmission company is not clearly defined with regard to the issue of Y2K readiness of the substation equipment. Some of the MNGP equipment resides in the substation, and because of this, the question of who should perform the Y2K assessment is not yet resolved. The equipment in question includes metering and relaying equipment. The corporate level bi-weekly project team meeting, which includes MNGP Y2K project management, is addressing this issue.

3.0 AUDIT TEAM FINDINGS

The following six observations were made by the audit team of the MNGP Y2K project:

  1. The licensee's MNGP Year 2000 Readiness Implementation Plan is a comprehensive document and is based on the guidance contained in NEI/NUSMG 97-07 with additional plant-specific procedures for evaluation of computer software and embedded software. The plan and associated procedures make use of existing plant procedures for software configuration control, software quality assurance (QA), software V&V, and change reporting. The plan is implemented through a project team consisting of a PM and technical specialists. The assessment and evaluation process requires the interaction of a cross-section of the plant organization.
  2. The MNGP Year 2000 Readiness Project has the support of a senior management sponsor. At present, communication of the progress of the project to senior management is through a project tracking report. Once the project's initial assessment is completed (scheduled for November 1998), bi-monthly project status meetings with NSP corporate senior management are planned.
  3. The audit team was under the impression that all nuclear power plant licensees had started their facility-specific Y2K program by early 1998 because NEI/NUSMG 97-07 was provided to senior utility management in November 1997. The MNGP Year 2000 Readiness Project was formally started in June 1998 and incorporated into the NSP corporate Y2K program at that time. The licensee was aware of the Y2K problem in late 1996 and had initiated an ad-hoc evaluation of some MNGP computer systems (e.g., plant process computer, plant security computer, and the turbine electronic pressure regulator) in 1997. The MNGP project is at the initial assessment stage now which is expected to be completed by October/November 1998. The overall MNGP Y2K project is scheduled to be completed by July 1999 with readiness achieved at that time. The audit team considers the schedule to be an ambitious one. However, the licensee appears to be able to meet the project schedule since (1) the number of software items at the site that are to be assessed for Y2K vulnerabilities (290 software items and around 500 embedded components per initial inventory) is not large, and (2) the licensee appears to have already identified and begun upgrades to major critical computer systems and components for Y2K compliance/readiness, and (3) licensee participation in BWR Owners Group and utility alliance efforts is permitting a more rapid assessment and remediation of systems and equipment because of information sharing than if the licensee had to proceed on its own. The audit team notes that detailed assessment, including some testing and remediation, and subsequent associated testing of some remaining critical systems and components are major tasks yet to be done.
  4. The audit team had planned to review the outline of the licensee's Y2K contingency plan for MNGP. However, the licensee has not as yet started on the plant Y2K contingency plan. The projected start date for MNGP Year 2000 Contingency Plan is January 1999. The Y2K PM stated that the contingency plan will be based on the guidance in NEI/NUSMG 98-07 and initiated in parallel with the detailed assessment efforts of the overall MNGP Y2K project. The audit team pointed out that a single point-of-contact for contingency planning has not been identified in the existing project team. The audit team believes that completion of the detailed Y2K contingency plans at MNGP can be achieved by July 1999 with the necessary attention provided by the Y2K PM and senior management.
  5. NSP corporate efforts and interfaces with its generation Y2K projects, including MNGP and Prairie Island Y2K projects, are good for addressing electrical grid reliability and availability issues . The audit team notes that the biweekly project team meeting is a good vehicle for identifying and assigning responsibilities for interface items that might affect plant operations and grid concerns such as the substation equipment issue noted above.
  6. According to the licensee's present plan, MNGP is to be shut down for reactor refueling in December 1999. However, there is a possibility that the unit may continue to operate during the December 31, 1999 - January 1, 2000, roll-over period. The NSP corporate Y2K program and MNGP Year 2000 Readiness Implementation Plan and associated contingency plans will consider both MNGP operating conditions. 
    Date:  October 1998

Table 1   MNGP Y2K Project Implementation Schedule
Table 2   Software Inventory
Table 3   Inventory of Safety-Related Software at MNGP
Table 4   Inventory of Embedded Systems
Table 5   Embedded Components Reviewed by the Audit Team
Table 6   Safety-Related Embedded Components Identified by
MNGP

Attachment 1   Y2K Readiness Audit Plan
Attachment 2   Documents Reviewed
Attachment 3   Entrance Meeting - Attendees