Note: The 235-XXX-XXX manuals do not provide maintenance procedures for the repair of equipment (for example, tape drives or disk drives) manufactured by vendors other than AT&T. To identify the appropriate maintenance manual for each unit of such vendor equipment, refer to Section 3 of AT&T 235-001-001, Documentation Description and Ordering Guide. This guide only provides the vendor's address associated to the unit in question, not the procedures used to repair the faulty equipment. This document describes the maintenance concepts and built-in maintenance capabilities of the 5ESS(R) switch. The information is necessary to perform system evaluation and maintenance. To be adequately prepared to maintain the 5ESS switch, the following prerequisites are recommended for maintenance personnel: a. Must be familiar with telephone equipment and terminology. b. Must complete maintenance training on the 5ESS switch. c. Must be familiar with the information in the following manual(s): o AT&T 235-100-125, System Description 5E6 and Later o AT&T 235-900-106, Product Specification 5E6 o AT&T 235-900-107, Product Specification 5E7 o AT&T 235-900-108, Product Specification 5E8. o AT&T 235-900-109, Product Specification 5E9(1). Note: These manuals also contain a section concerning Environmental Specifications (physical specifications) that can be useful to maintenance personnel. Product rating for the 5E2(2), 5E3, 5E4, and 5E5 software releases has been changed to Discontinued Availability (DA). Effective with Issue 7.00, all information on the DA rated software releases is being removed from this document. This document is updated to provide coverage for the 5E7 (Issue 5.00), 5E8 (Issues 6.00 and 6.01), and 5E9(1) (Issue 7.00) software releases. Sections .RM 1.2.2/, .RM 1.2.3/, and .RM 1.2.4/, respectively, contain update information on these software releases. Note: The 5E2(2) through 5E5 information is removed in Issue 7.00 of this document. (See statement relative to DA rating of software releases at the beginning Section .RM 1.2.1/.) For the 5E7 software release, Issue 5.00 of this document was updated for the following reasons: o Removed information on the 5E2(1) software release from all sections. [All 5E2(1) offices have been retrofitted to a later software release.] o Updated Table .AW TA/. o Changed information on the Call Monitor feature is Sections 2 and 3. o Updated information on the Office Data Base Editor (ODBE) is Section 3. o Updated information on the Access Editor (ACCED) is Section 3. o Updated information on the Current Update Data Base and History File Editor (UPedcud) is Section 3. o Made screen corrections, added screen abbreviations, and added command descriptions on Trunk and Line Work Station (TLWS) Task Selection Pages for 5E2(2) through 5E6 software releases in Section 3. o Added information on TLWS Task Selection Pages for the 5E7 software release in Section 3. Note: Information on TLWS task selection pages and commands is completely rewritten in Issue 7.00 of this document. o Reorganized and reformatted information on Generic Utilities to present information on this tool in a more descriptive format as opposed to the redundant Input/Output message data included in previous issues. o Added the master control center (MCC) Page Location Guide to the Introduction subsection in Section 4. Note: In Issue 7.00, the title of the Introduction subsection is changed to Introduction to MCC Pages. o Updated, added, and deleted text and screen displays in Subsections 5E2(2) through 5E6. These updates, additions, and deletions were the result of in-depth reviews for accuracy of all MCC page displays and associated text by BTL developers. o Added the 5E7 Subsection to Section 4. The 5E7 Subsection contains the MCC page displays that changed with the 5E7 software release. o Made minor corrections to text, tables, and figures throughout the document. Note: The 5E2(2) through 5E5 information is removed in Issue 7.00 of this document. (See statement relative to DA rating of software releases at the beginning Section .RM 1.2.1/.) For the 5E8 software release, Issues 6.00 and 6.01 of this document were updated for the following reasons: o Updated Table .AW TA/. o Updated information on Software Release Retrofit/Update/Large Terminal Growth in Section 2. o Updated information on Access Editor (ACCED) in Section 3. o Added information on Common Network Interface Data Base Consolidator (CNIDBOC) in Section 3. o Updated information on BROWSE in Section 3. o Updated information on Generic Access Package (GRASP) in Section 3. o Updated information on the Operation Support System (OSS) Routine Exercises (REX) Scheduler Program in Section 3. o Added information on the Automatic REX Scheduler in Section 3. o Added information on the 108-type test line in Section 3. o Added information on basic rate interface (BRI) access to the 108-type test line in Section 3. o Added information on Trunk and Line Work Station (TLWS) Task Selection Pages for the 5E8 software release in Section 3. Note: Information on TLWS task selection pages and commands is completely rewritten in Issue 7.00 of this document. o Updated the Generic Utilities information in Section 3 to add commands for two new 5E8 processors, Integrated Digital Carrier Unit (IDCU) and Integrated Digital Carrier Unit Loop Side Interface (IDCULSI). o Updated information in Table .AW TAD/, Directory of Service Centers. o Updated information in Table .AW TAE/, Directory of Service Support Organizations. o Updated the master control center (MCC) Page Location Guide in the Introduction subsection of Section 4. The update included adding all MCC pages in the 5E8 subsection. Note: In Issue 7.00, the title of the Introduction subsection is changed to Introduction to MCC Pages. o Updated the general description of the 1000 SM Page Index page in 5E2(2) through 5E7 subsections. o Updated information on MCC Page 105/106 in the 5E2(2) subsection. o Updated information on MCC Page 1950 in the 5E5 subsection. o Updated information on MCC Pages 115 (CM2 version), 116, and 1950 in the 5E6 subsection. o Updated information on MCC Pages 110, 116, 123, 125, 1800,X, 1850, and 1851 in the 5E7 subsection. o Removed information on MCC Page 1950 from the 5E7 Subsection. (The update to the 1950 page in the 5E6 Subsection applies to 5E6 and later software releases.) o Added the 5E8 Subsection to Section 4. The 5E8 Subsection contains the MCC page displays that were added with or changed with the 5E8 software release. o Made minor corrections to text, tables, and figures throughout the document. This document is updated to Issue 7.00B, May 1994, to add the results of call failures in Section .RM 3.5.1.10/, Call Monitor Reports. This document is updated to Issue 7.00A, December 1993, for the following reasons: o To remove the requirement for circuit packs to be tested on site o To add the DGR state to MCC page 118 o To update information on MCC pages 1521,XX and 1522,XX,Y. o To add a note for the user to insert RC/V view 8.3 if it does not exist. For the 5E9(1) software release, Issue 7.00 of this document is updated for the following reasons: o Update Table .AW TA/. o Update information on the Call Monitor in Sections 2 and 3. o Update Single Process Purge and Selective Initialization information in Section 2 to include wideband calls. o Add information on the automatic trunk test scheduler (ATTS) in Section 3. o Update Trunk and Line Work Station (TLWS) information in Section 3 for 32 test positions in 5E9(1) and to include test position resources and resource assignment. o Update line and trunk testing information in Section 3 to include wideband calls. o Change the format of information on TLWS Task Selection Pages in Section 3 to eliminate redundancy and reduce the number of text pages. o Add 5E9(1) examples of all TLWS pages to reflect new page layout and command changes and add information on new 5E9(1) Pages 5000,3 (line), 5000,3 (trunk), 8000, 9000,1, 9000,2, and 9201. o Add information on input message editing and history in Section 3. o Add information on the Ring Generic Access Package (RGRASP) in Section 3. o Add information on the Automated Circuit Pack Return Tag (RTAG) tool in Section 3. o Add information on the Automated Static ODD (SODD) Audit in Section 3. o Update the master control center (MCC) Page Location Guide in the Introduction to MCC Pages (formerly Introduction) subsection of Section 4. The update includes removing references to the 5E2(2) through 5E5 software releases and adding references for all MCC pages included in the 5E9(1) subsection. o Remove subsections 5E2(2) through 5E5 in Section 4 and move to the 5E6 subsection all MCC page displays valid for 5E6 (or 5E6 and later) that were previously included in subsections 5E2(2) through 5E5. o Add the 5E9(1) subsection to Section 4. The 5E9(1) subsection contains MCC page displays that were added with or changed with the 5E9(1) software release. o Make minor corrections to text, tables, and figures throughout the document. The information in this document is applicable to all switches equipped with 5E6 and later software releases. When text applies to a specific software release, the applicable software release is indicated. When new software releases are developed that affect this document, updates will be made. Also, this document is utilizing an issue number. The overall structure is outlined as follows: 1. Section 1--Introduction: This section summarizes the type of information contained in the document, gives the purpose of this information, and defines its organization. In addition, this section identifies the three maintenance manuals provided for maintenance and explains the function of each. 2. Section 2--Maintenance Philosophy: This section describes the following maintenance capabilities of the 5ESS switch: o Maintenance overview. o Remote maintenance capabilities. o Central office maintenance plan. o System evaluation and maintenance stimuli. o Maintenance tasks: a. Scheduled routine maintenance b. Nonscheduled routine maintenance c. Corrective maintenance tasks d. System recovery e. Operator Services Position System (OSPS) maintenance f. Maintenance of vendor equipment. o Line unit trouble clearing guide. o Trunk and line maintenance: a. Per-call tests b. Routine line and trunk tests c. Tests provided d. Call monitor. o Recent change (RC), field, and software release retrofit/update. o Change notices (CN). 3. Section 3--Maintenance Tools: This section describes the following maintenance tools available for use in the 5ESS switch: o Display administration process (DAP)/Non-DAP terminal. o MCC. o Supplementary trunk and line work station (STLWS). o TLWS. o Trunk and line maintenance. o Test access unit (TAU). o Directly connected test unit (DCTU). o Remote office test line (ROTL). o TLWS task selection page displays. o Recent change/verify (RC/V). o Screen program user's guide. o How to use input/output (I/O) messages. o Office data base editor (ODBE). o Access editor (ACCED). o Common network interface data base consolidator (CNIDBOC). o Automated circuit pack return tag (RTAG) tool. o Software debugging and troubleshooting tools. o Generic utilities. o System log files. o Diagnostics: -- Diagnostic types -- Diagnostic input/output messages -- Routine exercises (REX) -- Operation Support System (OSS) REX scheduler program -- Automatic REX scheduler. o How to use switching module (SM) peripheral Fault Recovery Message (Verbose Mode). o Routine tests. o How to use office backup schedules. o Circuit pack handling procedures. o Spare circuit packs. o Circuit pack repair service and return procedures. o Dynamic Audits. o Static Audits. o How to use program record and program map documents. o How to use program change document, symbol address cross-reference index, and function address program record (PR) name cross-reference index. o TR303 Integrated Digital Carrier Unit (IDCU) remote terminal provisioning. 4. Section 4--MCC Page Display: This section contains a detailed description of the page displays of the 5ESS switch MCC video terminal and the MCC Page Location Guide which can be used to locate specific page displays for specific 5E6 and later software releases. Section 4 is divided into five subsections as follows: o Section .RM 4.2/: Covers the introduction to the MCC page displays o Section .RM 4.3/: Covers the 5E6 software release o Section .RM 4.4/: Covers the 5E7 software release o Section .RM 4.5/: Covers the 5E8 software release o Section .RM 4.6/: Covers the 5E9(1) software release. Since this document has been developed to describe maintenance concepts and maintenance capabilities for the 5ESS switch, it is appropriate to identify the manuals that contain the procedures used to maintain the switch. 1. AT&T 235-105-210, Routine Operations and Maintenance Procedures: Contains the descriptive material and detailed procedures for routine operations and maintenance of the 5ESS switch. The following is a list of the sections in this document: o Equipment Test List (ETL) o Operations (system control functions) o Memory Alteration Description o Memory Alteration Procedures o Abnormal Input Message Acknowledgments o Fan and Alarm Tests o Moving Head Disk (MHD) Procedures o Miscellaneous Routine Procedures o ROTL o Routine Exercise Procedures. The AT&T 235-105-210 also has a job aid for O&M Checklist which must be ordered separately (see AT&T 235-001-001, Documentation Description and Ordering Guide). Note: Refer to Table .AW TA/ for a complete list of the operation and maintenance procedures included in AT&T 235-105-210. 2. AT&T 235-105-220, Corrective Maintenance Procedures: Contains three sections as follows: o Hardware - Maintenance Procedures: This section contains a series of task-oriented corrective maintenance procedures that can be used by personnel who are involved in maintaining various hardware units and circuits of the 5ESS switch. Also, some procedures are used to resolve subscribing customer service complaints. o Office Dependent Data - Maintenance Procedures: This section contains a series of task-oriented corrective maintenance procedures that can be used to maintain and repair office dependent data (ODD) associated with the switch. o Supporting Information: This section contains a tabular list that describes all the diagnostic phase descriptions and a Basic Rate Interface (BRI) trouble-shooting diagram. The AT&T 235-105-220 also has a group of job aids for the TLWS poke commands which must be ordered separately (see AT&T 235-001-001, Documentation Description and Ordering Guide). Note: Refer to Table .AW TA/ for a complete list of the operation and maintenance procedures included in AT&T 235-105-220. 3. AT&T 235-105-250, System Recovery: Contains the descriptive material and detailed procedures of the software and hardware recovery capabilities of the 5ESS switch. Both automatic and manual recovery capabilities are covered. The following is a list identifying what is covered in this manual: o System Recovery Description: In the 5ESS switch, system recovery uses the concept of a network of independent processors to localize recovery actions. The major processors involved are the administrative module (AM), communication module processor (CMP) (added in the 5E6 software release), and the individual SMs. When a fault exists, fault recovery attempts to reconfigure the system to provide full system service (primarily by excluding the faulty unit). Several levels of recovery are available, and the system can automatically escalate to higher and broader levels if initial attempts fail. The higher recovery levels often include processor initializations. This section describes the various recovery levels (and their impact) when used in the different processors. The strategy of reconfiguration and escalation to higher recovery levels is also covered, as is the mapping between manual commands and internal recovery levels. o System Recovery Procedures: The procedures provided in this section are used to clear system failures which prevent the 5ESS switch from restoring itself automatically. Also, procedures are provided for analyzing the AM and SM initializations. Note: Refer to Table .AW TA/ for a complete list of the operation and maintenance procedures included in AT&T 235- 105-250. 4. AT&T 235-105-119, Maintenance Guide Utilizing OMS5: This document provides information related to utilization of the OMS5 program. The OMS5 program summarizes the receive-only printer (ROP) output into a readable format. This program provides the maintenance personnel with an easy way to analyze how the office is performing. After analyzing the OMS5 program summary, the maintenance personnel should use the Corrective and Routine Maintenance Manuals (listed previously) with this manual to correct faults that occur in the switch. The OMS5 program runs on the switching control center (SCC) minicomputer. The tape that contains the OMS5 program can be obtained via the order number AT&T 235-105-120. This maintenance guide and the OMS5 program can only be used via the SCC. The producers of this manual are constantly striving to improve quality and usability. Please use the enclosed user feedback form [REF. .RM 1.9/] for your comments and to advise us of any errors. If the form is missing or your comments will not fit, you can write to the following address: AT&T NETWORK SYSTEMS Quality Department 2400 Reynolda Road Winston-Salem, NC 27106 Please include the issue number and/or date of the manual, your complete mailing address, and telephone number. We will attempt to answer all correspondence within 30 days, informing you of the disposition of your comments. You may also call our Documentation HOT LINE if you need an immediate answer to a documentation question. This HOT LINE is not intended to eliminate the use of the user feedback form, but rather to enhance the comment process. The HOTLINE number is 1-800-334-0404 and it is available from 7:30 a.m. to 4:30 p.m. Eastern time (within North Carolina, dial 1-910-727-6681). Outside of those hours, the line is served by an answering machine. You can leave a message on the answering machine and someone will return your call the following business day. Also, document users who have access to UNIX(R) system electronic mail facilities may send comments via electronic mail. The electronic address is att!wrddo!hotline5. Please make sure that the document title, number, and issue number are included in the mail along with the sender's name, phone number, and address. This manual is distributed by the AT&T Customer Information Center in Indianapolis, Indiana. Most operating telephone companies should place orders through their documentation coordinator. Some companies may allow customers to order directly from the Customer Information Center; however, the majority do not. Companies that use documentation coordinators to manage their orders receive a significant discount. If you do not know the name/number of the documentation coordinator for your company, you may call 1-800-432-6600 to obtain the name and telephone number. Customers not represented by a documentation coordinator and AT&T employees can order the documentation for the 5ESS switch directly from the AT&T Customer Information Center. Proper billing information must be provided. These orders may be mailed to: AT&T Customer Information Center Order Entry 2855 N. Franklin Road Indianapolis, IN 46219 Orders may also be called in on 1-800-432-6600 or faxed in on 1-317-322-6484. Technical assistance for the 5ESS switch can be obtained by calling the Regional Technical Assistance Center (RTAC) at 1-800-225-RTAC. This telephone number is monitored 24 hours a day, 7 days a week. During regular business hours, your call will be answered by your local RTAC. Outside of normal business hours, all calls will be answered at a centralized technical assistance center where service-affecting problems will be dispatched immediately to your local RTAC. All other problems will be referred to your local RTAC on the next regular business day. How Are We Doing? \ \ Document Title: SYSTEM MAINTENANCE REQUIREMENTS AND TOOLS Document Number: AT&T 235-105-110 Issue Number: 7.00 Publication Date: November 1993 AT&T welcomes your feedback on this document. Your comments can be of great value in helping us improve our documentation. 1. Please rate the effectiveness of this document in the following areas: 4 ____________________________________________________________________ | | | | | | Not | | | Excellent | Good | Fair | Poor | Applicable | |______________________|___________|______|______|______|____________| | Ease of Use | | | | | ///////// | |______________________|___________|______|______|______|____________| | Clarity | | | | | ///////// | |______________________|___________|______|______|______|____________| | Completeness | | | | | ///////// | |______________________|___________|______|______|______|____________| | Accuracy | | | | | ///////// | |______________________|___________|______|______|______|____________| | Organization | | | | | ///////// | |______________________|___________|______|______|______|____________| | Appearance | | | | | ///////// | |______________________|___________|______|______|______|____________| | Examples | | | | | | |______________________|___________|______|______|______|____________| | Illustrations | | | | | ///////// | |______________________|___________|______|______|______|____________| | Overall Satisfaction | | | | | ///////// | |______________________|___________|______|______|______|____________| 2. Please check the ways you feel we could improve this document: [] Improve the [] Make it more concise/brief overview/introduction [] Add more step-by-step [] Improve the table of procedures/tutorials contents [] Add more troubleshooting information [] Improve the organization [] Make it less technical [] Include more figures [] Add more/better quick reference [] Add more examples aids [] Add more detail [] Improve the index Please provide details for the suggested improvement._________________ ______________________________________________________________________ 3. What did you like most about this document? ______________________________________________________________________ ______________________________________________________________________ 4. Feel free to write any comments below or on an attached sheet. ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ If we may contact you concerning your comments, please complete the following: Name: ______________________________ Telephone Number: (___)__________ Company/Organization: ___________________ Date: ______________ Address: ______________________________________________________________ When you have completed this form, please fold, tape, and return to the following address or Fax to: 910-727-3043. DOCUMENTATION SERVICES 2400 Reynolda Road Winston-Salem, NC 27199-2029 The following is a list of manuals that the user should be aware of when reading this document: o AT&T 235-600-700, 5ESS Switch Input Messages Manual o AT&T 235-600-750, 5ESS Switch Output Messages Manual o AT&T 235-105-119, Maintenance Guide Utilizing OMS5 o AT&T 235-105-210, Routine Operations and Maintenance Procedures o AT&T 235-105-220, Corrective Maintenance Procedures o AT&T 235-105-250, System Recovery o AT&T 235-600-104, Translations Data 5E6 o AT&T 235-600-105, Translations Data 5E7 o AT&T 235-600-106, Translations Data 5E8 o AT&T 235-600-107, Translations Data 5E9 o AT&T 235-600-400, Audits Manual o AT&T 235-600-500, Asserts Manual o AT&T 235-600-510, Software Analysis Guide o AT&T 235-600-601, Processor Recovery Messages o AT&T 235-900-106, Product Specification 5E6 o AT&T 235-900-107, Product Specification 5E7 o AT&T 235-900-108, Product Specification 5E8. o AT&T 235-900-109, Product Specification 5E9(1). Note: Other manuals not identified previously that can be helpful to the customer are listed in AT&T 235- 000-000, Numerical Index - Division 235. This index is for informational purposes only. Site documents should be ordered from the applicable H- drawing. If you do not have a copy of this index, it can be ordered from the Customer Information Center in Indianapolis, Indiana. The 5ESS(R) switch is supported by many built-in maintenance aids: o Simplified human interface system o System status displays o Combined audible and visual alarm system o Automatic routine testing of circuitry o Routine exercise (REX) capability o Automatic fault detection and recovery o Manual testing capability o Remote maintenance capability o Duplication of common control equipment o Compact physical size. The MCC is the primary communication link between on-site maintenance personnel and the 5ESS switch. The MCC provides the interface capability for administrative and maintenance tasks. The major components of the MCC (Figure .AW G1/) consist of the following: o A video display terminal with keyboard o A receive-only printer (ROP) (optional) o A key telephone set with loudspeaker o A test access unit (TAU). Page displays on the video display terminal provide the status and control information needed to perform maintenance tasks. Maintenance requests are input using the keyboard. The ROP provides a hardcopy of input and output messages. Thus, a record is available for future reference. The key telephone set is used to communicate with work areas outside the office. This telephone set can be used independently of the 5ESS switch, thereby ensuring outside communication if an office outage occurs. A loudspeaker is provided for communication at times when maintenance personnel need the use of both hands. The TAU provides telephone jacks that enable communications with other work areas in the office. In addition, TAU provides access to trunks and lines for maintenance activities. The real-time system status is shown at the MCC video terminal on the second and third lines of the page display. (See Figure .AW G2/.) Thus, the occurrence of any abnormal conditions is displayed immediately. The MCC status display must be valid at all times. This requires monitoring of the time indicator to ensure reliable display indications. The time-of-day indicator (24-hour clock) at the top right of the video display is incremented every 5 seconds. Observation of this time indicator helps to determine if the display interface is operating. If the indicator is not changing, the interface is not working, and the entire video display is invalid. Figure .AW G2/ is an example of the MCC page display design. Whenever an alarm condition occurs, an audible/visual alarm is activated to ensure that maintenance personnel are informed even if the MCC terminal is not being monitored. To make it easier for maintenance personnel to quickly locate off-normal conditions on the video displays, various video attributes such as reverse video, flashing, intensity, and color (optional) are used in addition to text. The particular combination of these attributes depends on the maintenance ``state.'' Refer to Table .AW TAG/ for additional information on the most commonly used MCC states and their video characteristics. When an alarm condition occurs, the severity of the alarm is indicated by the level indicator (CRITICAL, MAJOR, or MINOR) backlighting in the summary status area at the top of each display. Each alarm level (CRIT, MAJ, or MIN) also has its own distinct sound. The unit indicator that represents the particular area of alarm condition (for example, OVERLOAD or BLDG/PWR) flashes until the alarm is retired. To retire the audio/visual alarms, depress the ALM RLS function key. Once the alarm is retired, the level indicator returns to normal video and the unit indicator stops flashing. The alarm level color remains until the MCC page associated with the unit has been displayed. At this point, the unit indicator goes to black on cyan which indicates the problem still exists but is being investigated. When the condition causing the alarm is corrected, the unit indicator returns to normal video. Automatic routine tests are checks and tests run automatically on a prescheduled basis to verify the integrity of a circuit, a unit, or the entire system. The system has built-in self-checks which constantly check parity, framing, and protocol of words and messages. In addition, periodic routine exercises are run. These exercises verify the complete integrity of all units including both the operational and the maintenance-related parts of units. Per-call tests are run on a line every time it is used. Table .AW TB/ provides a list of automatic routine tests, intervals, and functions. Audits are also run to check software programs. Audits are run periodically and during slack call processing periods. The Call Monitor makes test calls for periodic analysis to detect the loss of call processing functionality. The MCC enhances the manual testing capabilities of the system. Diagnostics can be run using the keyword at the MCC to input the appropriate messages. If the appropriate input message is not known, the user can enter the dgn:keyword?, where the keyword for example could be ``luhlsc'' and the symbol ``?'' indicates help when an appropriate input message is desired. Figure .AW G3/ is an example of how the help command can be used. The first help command gives the complete input message, and the second help command gives the options associated with the input message. User's actions are indicated in bold type. In addition, many block diagram-type page displays (that is, MCC page displays -- see Section .RM 4.2/) are available to help in locating and identifying faults. The TAU provides a means of connecting test equipment to various lines and trunks. Direct connection and terminal access jacks are contained in the TAU. Section .RM 3./ contains a description of TAU. The duplication of common control equipment permits switching from a faulty unit which is active to a standby unit. Thus, the faulty unit can be taken out of service (OOS) and repaired without impairing the switching capability of the office. Maintenance of the system can be performed remotely by operational support systems (OSS), located at the remote switching control center (SCC). The MCC human interface capabilities are available at the remote SCC. This includes the video display, input/output, and alarm control capabilities. The trunk and line work station (TLWS) TAU is not provided at the remote SCC. The objective of this maintenance plan is to identify both hardware and software faults in the 5ESS switch before they reach a service-affecting level. If this maintenance plan is followed, customer complaints can be reduced to a minimum and be a useful indication of the switch performance. If customer complaints suddenly increase in a normal office, the complaints can provide information useful in identifying problems and their causes. Therefore, customer complaints can help identify the following: Complaint location: o Switching module (SM) or group of SMs o Remote switching module (RSM) o Line unit (LU) or group of line units. Complaint type: o Plain old telephone service (POTS) o Private branch exchange (PBX) o Features o No dial tone o Call cutoff. Complaint cause: o Software update o Change notice (CN) Application o Unusual office activity o Cable cut o Nondiagnosable hardware fault. If unusual or unresolved conditions and complaints cannot be resolved at the local level, contact your next level of technical support. Until such time as the customer complaint rate is under control, the number of customer complaints received is not an accurate indication of switch performance. A more accurate accounting can be achieved through the monitoring of the operational test failures (OTFs) and the grid fabric failures. ROP Analysis: Periodic checks should be made using the daily reports to determine whether or not hardware faults are occurring. AT&T 235-105-220, Corrective Maintenance Procedures, can be used to identify the hardware involved in the following instances (except for the line removal reports; they are covered in this manual): o Unit diagnostic (routine exercise) failure o Grid fabric failures o OTFs -- Set the verbose mode periodically to allow printing of these messages o Per-call test failures (PCTFs) o Machine detected interoffice irregularities (MDIIs) o Line removals. Action: If any of the previous conditions occur, then go to the appropriate section in this manual for a more complete description and the appropriate action. Note: If you are unable to resolve the errors after referencing the section and documents mentioned, then contact the next level of technical support. ROP Analysis: Periodic checks using AT&T 235-070-100, Traffic and Plant Measurements, Appendix 1 of Administration and Engineering Guidelines, should be made to detect if large numbers of the following types of messages have occurred: o Manual action asserts o Audits o Interrupts [for example, single-process purges (SPP) is a first-level interrupt]. Action: Set the print log status periodically to print these types of messages. Upon receiving a repetition of these, refer to AT&T 235-600-500, Asserts Manual, and AT&T 235-600-400, Audits Manual, for the proper action to take. Note: If you are unable to resolve the errors after referencing the document mentioned, then contact the next level of technical support. Remember, initializations can cause codes 5, 7, 8, etc., customer complaints, so frequently check the ROP for indications of trouble. For more information on initializations, refer to AT&T 235-105-250, System Recovery, which has some general information about system recovery/initialization. Office Backup Methods: There are four levels of backup for the 5ESS switch disk drives. These levels are as follows: a. Memory to primary disk backup b. UNIX(R) Real-Time Reliable (RTR) operating system root partition to backup-root partition backup c. Office dependent data (ODD) backup to tape d. Full office backup to tape. For detailed procedures covering the disk drives, refer to the Memory Allocation Procedures in AT&T 235-105-210. For information on Backup Schedules and Guidelines, refer to Subsection .RM 3.23.7/ in this manual. Program update is the process that activates orderly program changes in the switching equipment software. The changes are made to solve a system problem. The following two types of program updates are available: o Official Fixes: Software updates o Emergency Fixes: Temporary measurement plan (TMP) software updates. In-service offices receive most software fixes via software updates. The following list of operations should be adhered to when applying software updates: 1. In a post-turnover office, when it is not in a retrofit or restart mode, the Operating Telephone Company (OTC) is responsible for obtaining and applying all applicable software updates. 2. Maintain software updates to the current Software Change Administration and Notification System (SCANS) level. 3. Follow the software update's special and soak instructions exactly. In an in-service office, remember to SOAK all software updates for at least a full day. 4. Some software updates require a boot of the administrative module (AM) or a craft initialization in order to make portions work properly; therefore, the application of the software update should be carefully monitored by the Switching Control Center System (SCCS) or site personnel until it is complete. 5. When a software update requires a boot, always perform the boot during a low-traffic period. 6. If a software update fails during application (apply, soak, and/or official), and the situation cannot be resolved, escalate to the next level of technical support. Warning: DO NOT remove files such as ``Cud'' and ``.pupstat,'' or any file in ``/etc/bwm'' without first consulting with Electronic Switching Assistance Center (ESAC), Regional Technical Assistance Center (RTAC), or Customer Technical Support (CTS) Organization. This section represents some preventive maintenance procedures needed to keep the 5ESS switch operating efficiently. The fan filters in the 5ESS switch frames and moving head disks (MHDs) should be checked periodically. A dirty filter will not allow adequate air to circulate within the equipment and could lead to early failure in the hardware. For information regarding replacement frequency and replacement procedures for both frame fans and MHD fans, refer to AT&T 235- 105-210, Routine Operations and Maintenance Procedures. Review the REX output messages daily to see if any failures have occurred. Normally, if a diagnostic failure occurs during REX, that hardware will remain OOS and must be manually diagnosed and restored to service. This is a very important action to conscientiously perform. If one side of a duplex unit fails REX diagnostics, it is not restored. If the other side of this unit takes a fault, there is no way to recover and the unit will be in duplex failure mode. It is easy to see how critical this would be if the unit was important to call processing. Check the ROP Reference Guide section of AT&T 235-105-119, Maintenance Guide Utilizing OMS5, for canceled or needed REORGANIZATION messages. Determine why the relation reorganization failed. If unable to locate the trouble or resolve it, escalate to the next level of technical support. All critical system status indications are provided locally and remotely. The MCC provides real-time system status summary indications, control and display capabilities, and human interface. These capabilities are also remoted to the remote switching control center. System status and alarm indications are displayed on the remote switching control center critical indicator panel. The status display provides a comprehensive presentation of system conditions including the following: o Alarms and other abnormalities o Abnormal load conditions o Significant control in effect o Individual unit status. The status display is made up of abbreviated name displays of each monitored unit or condition. Normal operating conditions are displayed in dynamic (light on dark) text. Trouble conditions are displayed in steady bold reverse video (dark letters on enlarged bright background) or a color combination. All alarm conditions are displayed by a unit indicator such as - AM, SM, and a level indicator - (CRITICAL, MAJOR, or MINOR). An audible indication is also sounded as follows: o In minor alarm conditions, the minor alarm horn sounds (if office is equipped). o For major alarm conditions, the audible alarm chimes at a slow rate. o For critical alarm conditions, the audible alarm chimes at a fast rate. There are two system alarm release modes: automatic alarm release and manual alarm release. If the system is in the automatic alarm release mode, the audible alarm and the flashing alarm conditions are released 5 seconds after initialization. If the system is in the manual alarm release mode, the audible alarm and flashing alarm conditions are released by operating the ALM RLS function key on the video terminal keyboard. Minor audible alarms are retired after 5 seconds in either mode. Released alarms and controls in effect remain in the alarm condition until the system has been restored to normal operating condition. The alarm release mode is changed via a maintenance command available on display Page 105/106, 116, or an input message. Table .AW TC/ lists MCC status indicators and their meanings. Note: A display administration process (DAP) terminal must be used to access the control and display portion of the MCC or remote switching control center video display. The DAP terminal can be used to perform any command that is needed to maintain the switch (for example, poke command 330 on MCC display Page 1240 restores MSGS 0 to service). These terminals are defined in the data base, and a number (office dependent) of DAPs can be assigned. A maximum of eight DAP terminals for 5E6 or sixteen DAP terminals for 5E7 and later can be used simultaneously per switch. Non-DAP terminals enable the user to perform the same functions as DAP terminals except for being able to access and display the MCC page displays. The non-DAP terminals use message mode commands to maintain the switch (for example, input message, RST:MSGS=0). The control and display portion of the MCC or remote switching control center video display is used to monitor at a subsystem level. Control and display consists of many separate pages that can be displayed individually. Each page shows the operating condition and a menu of possible input commands for a particular subsystem. Figure .AW G4/ shows a control and display page. The display conventions used for equipment status are also used for all control and display page displays. An index page is provided to allow quick access to any of the control and display pages during trouble situations. A message page is used whenever control and display information is not required (that is, the display of system status is all that is desired). This avoids confusion, since the display provides only the system status and input message information when the blank page is used. (Figure .AW G2/ shows the video display as it appears with the blank page display.) The input message area is used for inputting human interface messages. Refer to Section .RM 3./ of this document for details of how to use input messages. Any deviation from normal system operating conditions produces a printout on the MCC or remote switching control center. The printout contains all data relevant to the condition, diagnostic results, and a list (by priority) of suspected faulty circuit boards. Periodic traffic and plant summaries are also printed on the printer. All of these printouts are important in determining system status, with diagnostic information and circuit board lists being of primary importance to maintenance personnel. The printer should be connected whenever maintenance is being performed in the office. For detailed analysis of printouts, refer to AT&T 235-600-750, Output Messages Manual. Routine maintenance is performed on a specified schedule to secure maximum performance of the total network. Since peak load periods and features are different from office to office, some tests (for example, REX test) may not have specific test schedules that are best for all of the offices. In this situation, the equipment test list (ETL) can be very helpful in giving references to procedures and recommended time intervals to perform these types of tests. Refer to AT&T 235-105-210, Routine Operations and Maintenance Procedures, for the 5ESS switch routine maintenance schedules (residing in the ETL). More importantly, this manual contains descriptive and detailed procedures for the schedules listed in the ETL. Nonscheduled routine maintenance procedures are basically those procedures that are not listed on the ETL. The following list contains some of the nonscheduled operations: o System Control Functions: a. Loading automatic message accounting (AMA) tapes b. Set/change date and time c. Alarm system assignments. o Call Trace Procedures: a. Nuisance call trace b. Interoffice call trace c. Utility call trace. o Miscellaneous Routine Procedures: a. Bring up AMA teleprocessing system (AMATPS) b. Bring up central trunk test unit (CTTU) c. Bring up Engineering Administration Data Acquisition System (EADAS). o Memory Alteration Procedures: a. Request software update summary b. Obtain ODD backup schedule c. Load software updates from tape. For the complete list of nonscheduled routine operations, refer to AT&T 235-105-210, Routine Operations and Maintenance Procedures. The video display pages are used together with the system status display and ROP output messages to isolate a hardware trouble to a specific unit. Then the system's diagnostic and grid exercise programs are used to pinpoint the trouble to the specific circuit pack(s) as described in Section .RM 2.5.3.2/, Hardware Repair Procedure (following). Also, a simplified trouble location procedure is shown in Figure .AW G5/. Note: Periods of AM insanity may affect MCC display and functional capabilities. Automatic trouble location is an integral part of the diagnostic and grid exercise programs in the 5ESS switch environment. These programs are designed to test functions at the circuit pack (board) level. Therefore, most test failures can pinpoint the fault to a small number of circuit packs. The diagnostic and grid exercise programs maintain a list (by priority) of suspect circuit packs at each test point in the diagnostic. During execution, this list expands and contracts as testing shifts attention among the hardware. Upon the first failure, the current circuit pack list is converted to a suspected faulty equipment list containing location information and printed on the ROP. Combined use of this list and the frame and shelf-mounted designation strips provide direct access to suspect circuit packs. The trouble repair procedures in AT&T 235-105-220, Corrective Maintenance Procedures, should be used to replace the faulty circuit pack. A sample faulty equipment list printout is shown in Figure .AW G6/. For each circuit pack, the following information is given: o AISLE: Aisle equipment frame location within the office. o MODULE: Switching module number. o CABINET: Cabinet type and number. o CODE: Circuit pack number. o FORM: Equipment forms. A form can be one of two types as follows: a. Series number with carrier pack b. Issue number with micro code. o EQL: Equipment physical location [for example, 53-016 (where 53 = vertical distance, in inches, of the circuit above the floor and 016 = horizontal distance of circuit from LEFT edge of bay in 1/8-inch increments)]. A third field [53-016-XX, where XX = depth (in the unit) in 1/10- inch increments] is also included. o TYPE: Helper unit. Note: When a note (for example, 8) appears in this column, refer to the ``TLP (Trouble Locating Process) NOTE APPENDIX'' in AT&T 235-600-750, Output Messages Manual. o DGN: Diagnose. o LUHLSC: Line unit high-level service circuit. Another maintenance tool available to maintenance personnel for locating trouble is the utility call trace. Utility call trace allows users to do the following: o Snapshot the hardware path representing a call connection. o Trace all connections of a call. o Trigger a call trace with a utility break point. o Print the path of the call through the switch in diagnostic format on the ROP and show it on the MCC screen. Craft and/or maintenance personnel can also use utility call trace to trace test calls. For example, to troubleshoot customer complaints, a test call can be traced to or from the customer to see what hardware is in use. In the 5ESS switch, data base inconsistencies can be identified by asserts and audits. The results of the asserts and audits can be sent to the system log file and/or the ROP. Audits and asserts are used in the 5ESS switch to verify and check the validity of software data structures such as the ODD and equipment configuration data (ECD) in the AM. Data base repair procedures are provided in AT&T 235-105-220, Corrective Maintenance Procedures. Reconfiguration is necessary to prevent faulty hardware from affecting system processing. Equipment can be reconfigured either automatically or manually. Basically, reconfiguration consists of the following: o Appointing other hardware to assume functions of the faulty hardware o Removing traffic from the faulty hardware o Removing faulty hardware from service o Updating system status with appropriate alarms, indicators, printouts, etc. Since the 5ESS switch varies in size and equipment usage, the recovery procedures vary in complexity. The large office obviously has a wider range of reconfiguration possibilities since it contains more hardware. Fault recovery is performed at a subsystem level whenever possible. Only the fault-associated subsystem(s) is affected during recovery. When a hardware fault is identified, the system begins automatic recovery procedures. If the system is in good operating condition prior to the fault and the fault is not catastrophic, automatic recovery should restore normal processing. Automatic recovery performs all the reconfiguration and initialization processes necessary to correct the problem. Automatic recovery restores the system in the great majority of all faults. Manual reconfiguration is used in the repair of a unit following automatic recovery or if automatic recovery does not place the faulty unit OOS and restore processing. Most manual reconfiguration is done from the MCC or the remote maintenance center (if provided). There are several numeric input commands on the control and display pages that can be entered from the terminal keyboard. They are as follows: o REMOVE: This series of commands (2XX and 2XXX) reroutes traffic from the affected unit and places the unit OOS. o RESTORE: This series of commands (3XX and 3XXX) diagnoses the unit to determine if it is capable of operating. If diagnostic returns all tests passed (ATP) or conditional all tests passed (CATP), the unit will be restored to service. Otherwise, the unit is left OOS. o SWITCH: This series of commands (4XX and 4XXX) causes all traffic and processing to be switched to the mate controller. o DIAGNOSE: This series of commands (5XX and 5XXX) requests diagnostics to be run on specified unit(s). The unit remains out of service until manual restoral to service is requested. o FORCE: This series of commands (4XX and 4XXX) unconditionally forces operation to the desired unit and puts the mate unit OOS Forced Unavailable. All of these commands are entered conditionally, and the system enables them. The video page display for each unit has a menu of commands (and input codes) available for that unit. If the system is experiencing problems, it may not honor these input requests. Unconditional options and manual overrides are provided for these cases. The amount of direct control available depends on the type of unit involved. Fully duplicated (duplex) units [for example, message switch control unit (MSCU), office network timing complex (ONTC), and module controller/time slot interchanger (MCTSI)] contain control/display (CD) packs with several status light-emitting diodes (LEDs) and manual reconfiguration switches. The CD packs such as the SN412, SN516, and the TN1077 all contain four switches and five LEDs which provide and display the following capabilities: o ON: A momentary pushbutton switch used to power up the unit. o OFF: A momentary pushbutton switch used to power down the unit only if that unit is not in service or is unavailable. o RST/ROS: A rocker switch used to request a unit be restored to service or conditionally removed from service. o MOR: A momentary manual override switch. Whenever this switch is simultaneously depressed with the OFF switch, power is turned off regardless of the hardware state (ACT, STBY, or OOS). o OFF: A red LED that lights whenever unit power is off. o ALM: A red LED that lights whenever there is a power fault on the unit (fuse or converter alarm). Note that in the alarm state, all power may not be off in the unit. Once maintenance personnel power down the unit for repairs, the OFF LED lights and the ALM LED goes off. o OOS: A yellow LED controlled by the system. This LED lights whenever the unit is out of service. o RQIP: A green LED controlled by the system. This LED lights whenever a request to restore or remove a unit from service has been received by the system. If this request is denied or fails, this LED flashes for 5 to 6 seconds. The LED goes off when the requested action has been completed. o ROS: A green LED that lights whenever the restore/request out-of-service (RST/ROS) switch is in the ROS position. Figure .AW G7/ shows an example of the SN412/SN516 CD pack. Table .AW TD/ summarizes duplex control and display pack features for the 5ESS switch. Unduplicated (simplex) units are not equipped with control and display packs. All simplex requests (RST, ROS, etc.) are input via an input message. Simplex status [request in progress (RQIP, OOS, etc.)] is displayed and printed at the MCC or SCC. The AT&T 3B20D computer units are equipped with the TN5 AM C/D pack shown in Figure .AW G8/. This pack is equipped with an additional alarm cutoff/test (ACO/T) LED and switch that the SN412 and SN516 packs do not have. Unit controller conditions must be known at all times. This information is needed to define system status. Four general status states have been defined for the 5ESS switch unit controllers. They are active, standby, out of service, and unavailable. Table .AW TE/ summarizes basic controller states. At times, it is necessary to know how, or why, a controller entered a particular state. A set of state-qualifiers has been developed to further define a controller state. A combination of qualifiers may be required to specifically define a state. Table .AW TF/ shows qualifiers used and sample applications in the 5ESS switch. All duplex subsystems follow the same general methods of manual reconfiguration. Reconfiguration is accomplished by manual inputs at the MCC or remote maintenance center and/or the unit control and display circuit pack. Since simplex units are unique, they cannot be reconfigured as such. Therefore, circuit board replacement is the method of restoring simplex units. The other part of system recovery is initialization. Serious system difficulties may be caused by equipment (hardware) troubles, difficulties in executing the program (software), or by human error. Caution: If the system is failing to process calls properly (is not able to complete test calls, etc.), the system should be automatically attempting to recover itself by taking automatic emergency actions. This should be indicated to office personnel by status indicators, printouts, switching of system controllers, etc. If automatic emergency actions do not restore normal system processing and control, manual emergency actions or system recovery procedures will be required. The distributed processing architecture of the 5ESS switch employs many autonomous processors. The main processing units are the AM, the Communication Module Processor (CMP), and the individual SMs. Initialization can treat these processors both independently and collectively. Therefore, the following four types of initialization have been defined: o AM Initialization: This is an initialization of the AM. o CMP Initialization: This is an initialization of the CMP (added in 5E6). o SM Initialization: This is an initialization of the SM. o System Initialization: This is a complete initialization of the AM, the CMP, and the SMs. Note: Although slightly different actions can take place at each level in the AM, CMP (added in 5E6), and SMs, the overview of the philosophy and objectives of the initializations are the same. The various levels of recovery, their attributes, and recovery time estimates for individual SMs, the CMP, and the AM are explained in Table .AW TG/. In any case, the stimulus of an initialization is the failure of a check that indicates if the integrity of a processor or data base is questionable. Initialization is caused by a signal which is generated when the hardware or software detects an error (resets). It can also be caused by manually inputting initialization requests (interrupts) at the video terminal keyboard. An initialization consists of some or all of the following: o Restoring processor(s) to a good software state o Restoring the periphery to a good software state o Aborting certain activities o Zeroing or setting temporary data to a known good state o Reloading the memory from disk file. Not all of the preceding actions are performed on every initialization. An initialization can be more or less drastic depending on which and how many of the preceding routines are performed. The degree of initialization is determined by the system level count. The level count is incremented each time a recovery attempt fails within a predetermined time lapse. The higher the level count, the more drastic the recovery actions become. After an initialization occurs, an initialization timing interval exists for a short period of time. If no other initializations occur within this time interval, the level count will be reset to zero. For detailed information on initializations and recovery procedures for the 5ESS switch, refer to AT&T 235-105-250, System Recovery. This is the lowest level of software initialization. It is performed automatically in response to audit features, user program defensive check failures (asserts), and restarting from maintenance interrupts. Action associated with this level may be as follows: o Localized initialization of user-owned global data o Scheduling elevated audits o Logging failure information. Control flow is then returned to the previously interrupted point. The single-process purge (SPP) level is used whenever an error is detected which is severe enough to make it unsafe to return to the point of interrupt. The initialization action varies somewhat with the processing environment, but the primary objective is to restore a software configuration that can support resumption of normal processing. In general, the recovery actions associated with an SPP are as follows: o The running process or task is killed and, if essential, reinitialized. o Any global data being used by the process is restored. o Any hardware or software resources are recovered. o Any associated processes are informed. o Control is reestablished at a ``safe point.'' o Failure information is logged. Some recovery may be performed on a deferred basis by audits requested by the purge. In terms of call processing, if the current process is associated with a call, the call will be idled and the subscriber given reorder or dial tone. Wideband calls will be idled if the process is associated with a wideband call. Directed audits are used as an initialization action whenever inconsistencies are discovered in critical data structures such that continued operation is not possible. This level is generally invoked from either an audit or a user program defensive check failure (assert). The action taken is to audit, in an unsegmented mode, enough data to ensure that system operation can resume, and to schedule on a deferred basis any additionally required audit activity. Restart from a directed audit is generally via a single-process purge of the current process. Again, failure information is logged. If the running process was associated with a call, the call will be idled and the subscriber given reorder or dial tone. This is a full initialization of a single AM kernel process. All dynamic nonshared data is initialized or audited. All data shared among known processes is audited. In 5E6 and later software releases, common channel signaling is suspended during this short initialization. A selective initialization (SI) is a high-level initialization (although it is the lowest level processor-wide recovery action). This initialization can be performed automatically due to recovery actions or manually by maintenance personnel. The basic actions of an SI in the various processors are described as follows: o In the AM, an SI includes an unconditional bootstrap for all static text and data for both the UNIX RTR operating system and the 5ESS switch application processes. Then, all dynamic data is either initialized or audited (OOS hardware status and stable calls are preserved). Call processing functions in this processor are suspended during this time interval, but stable calls are preserved. o In the CMP (5E6 and later software releases), an SI does not include any hashsum checks or pumps. All dynamic data is either initialized or audited. Call processing functions are suspended during this interval, but stable calls are preserved. o In the SM, an SI does not include any hashsum checks or pumps. All dynamic data is either initialized or audited, preserving OOS hardware status and most stable calls (certain connections that would appear to be stable are disconnected due to their more complex dynamic data or ongoing message interaction with the switch, that is, 3- and 6-way calls and AP data links). Call processing functions within the initializing processor are suspended during the initialization interval. A manual SM selective initialization with an unconditional full pump is available (though it has additional affects on the SM of longer downtime and, possibly, a few lost stable calls due to the pump use of time slots). Note: Stable calls over SM selective initialization include wideband. Options to back out temporary recent changes and program updates are supplied when pumps are involved in the previous processors. A full initialization (FI) is the highest software level of processor-wide recovery actions. The FI can be performed automatically due to recovery actions or manually by maintenance personnel. There are several types of FI which can be used (for example, power up FI and FI with pump) each of which changes the severity of recovery action. Every FI will initialize hardware at some level even though it is mainly a software initialization. Different hardware levels will be seen depending on the processor being initialized. The basic actions of an FI in the various processors are described as follows: o In the AM, an FI includes an unconditional bootstrap of all static text and data for both UNIX RTR operating system and the 5ESS switch application processes. Then, all dynamic data is initialized. When the AM undergoes an FI, the hardware level selected can cause the loss of stable calls routed through the time multiplexed switch (TMS). During the initialization of the AM, the SMs are supplying dial tone and giving reorder to the customers or processing intra-processor SM calls if the stand-alone option is utilized. In 5E6 and later software releases, call processing will be maintained during an AM FI except at the highest hardware level, which reinitializes the TMS. New CCS calls will be inhibited until the CNI Ring is initialized. o In the CMP (5E6 and later software releases), an FI does not affect stable calls although any transient calls being processed at the time of the FI will be lost. The FI includes a conditional partial pump of any static text or data that fails to pass a hashsum check against a disk copy of the hashsum tables. All dynamic data is initialized. Both manual and automatic full CMP initialization with an unconditional pump can be invoked. o In the SM, an FI includes a conditional partial pump of any static text or data that fails to pass a hashsum check against a disk copy of the hashsum tables. Then, all dynamic data is initialized. Depending on the type of FI, an FI will also be performed on the SM's peripheral hardware and software. Any transient and stable calls being processed by the processor undergoing the full initialization or on connected peripherals will be lost. A manual SM full initialization with an unconditional pump is available. Options to back out temporary recent changes and program updates are supplied on any pump of the various processors. Office bring-up and dead-start situations use a manual system-wide FI (that is, AM, CMP, and all SMs). A postmortem dump is normally printed via the ROP to permit analysis of the cause of the system troubles; however, postmortem dumps can be logged in the postmortem dump log (PMLOG) or the DAYLOG file. The output message class (MSGCLS) is used by the maintenance personnel to specify where the postmortem dumps are to be sent, that is, to the physical device (ROP) or the software device (PMLOG or DAYLOG). Output consists of all data relevant to the trouble, such as the following: o Value of program counter at time of occurrence o Value of latched address and data bus o System process that last had control o Values of internal control registers o Stack area values of last system process o Contents of register area for last system process o Contents of all error registers which indicate the source of the error(s) o Contents of counters used for escalation to higher levels of initialization. An SM postmortem dump is normally sent to the ROP approximately 1 to 5 minutes after the system has gained sanity. The first report to indicate an SM initialization has been started is the high-level initialization report. The first line is as follows: REPT SM=a INITIALIZATION TRIGGER=bb EVENT=cc Refer to AT&T 235-600-750, Output Messages Manual, for details concerning the REPT SM message. Efforts should be made to analyze the cause of the initialization and to verify that the SM recovers properly. When the SM recovers, analyze the postmortem dump and any related error reports. The related reports will have the same event numbers. If the postmortem dump is not printed automatically, it is possible to obtain the postmortem dump by using the input command: OP:POSTMORT,SM=a [,OFL] [,SPP] [,EVENT] [,ESCAL] [,RCVY] [,DCF]; Refer to AT&T 235-600-750, Output Messages Manual, for details concerning the OP:POSTMORT,SM message. The postmortem dump report contains information about type and source of the interrupt that occurred in the SM controller and the resulting level of initialization. This report has two possible formats, the shortest of which is normally printed on the ROP, while the extended version is stored in the log file DAYLOG. The log file DAYLOG is kept in general to locate severe software faults. The contents of the file can be dumped by entering the following message for 5E6 and later software releases: OP:LOG:LG="DAYLOG"[:[DATE=b[&&c],][TIME=d[&&c],][KW="f",][ID=g,] [MSGCLS=l|TYPE=h,][LIMIT=i,][,CLASS=j|,DEVICE="k"]]; Refer to AT&T 235-600-750, Output Messages Manual, for details of this message. The short version of the previously mentioned SM postmortem dump reports has the following format: REPT SM=a,b HWLVL=c SWLVL=d e f g EVENT=h i j-ERR FAILING ADDR=k PROCESS:BG=r,s,t CM=u,v FG=w,x,y z [,TYPE=h] [,LIMT=i] [,CLASS=j|,DEVICE=k]] The extended version from the log file DAYLOG looks as follows: REPT SM=a,b HWLVL=c SWLVL=d EVENT=h i e f g j-ERR FAIL-ADDR=k l-m DATA-BUS=n TIME=o:p.q PROCESS: BG=r,s,t CM=u,v FG=w,x,y z ORIG-HW-STATUS: a : b c d a : b c d FINAL-HW-STATUS: a : b c d a : b c d PREVIOUS TYPE/COUNT: e f SHADOW TYPE/COUNT: g h AUX DATA: m n o p ESCALATION-COUNTS: i j k l Refer to AT&T 235-600-750, Output Messages Manual, for details of the REPT SM message. In addition to the postmortem dump reports, related error reports should be analyzed to locate the fault. Related error reports will have the same event number. Examples of related error reports are illustrated as follows: o INIT SM=a LVL=b SUMMARY EVENT=g ... o INIT SM=a,b LVL=c OSDS-M EVENT=f g o REPT SM=a DLI HW REGS EVENT=g ... o REPT SM=a SP HW REGS EVENT=b ... o REPT SM=a TSI HW REGS EVENT=c ... o REPT SM=a CI b HW REGS EVENT=c ... o REPT SM=a PI b HW REGS EVENT=c ... o REPT SM=a RLI b HW REGS EVENT=c ... o REPT SM=a MC b HW REGS EVENT=c ... Suppressing 5-Minute Automatic Dumps: The input command RLS:POSTMORT,SM=a; (MML format) may be used to suppress the 5- minute automatic dump. The command also clears the postmortem area; otherwise, the area will be cleared automatically 1 hour after the initialization. Error Source of Interrupt: In the report REPT SM=a,b HWLVL=c SWLVL=d e f EVENT=h i, field ``e'' reports hardware subunit which triggered the interrupt, and field `` f '' indicates the source of the error that caused the interrupt (see AT&T 235- 600-750, Output Messages Manual). A CMP postmortem dump contains information about the error(s) that caused the CMP to take recovery action. Information is sent to the ROP, usually within 1 to 5 minutes after the system has gained sanity, and is displayed in several types of output messages beginning as follows: REPT:CMP INIT:CMP REPT CMP= POSTMORT For additional information on these messages, refer to AT&T 235-600-750, Output Messages Manual. If the postmortem dump is not printed automatically, it can be requested via the following input message: OP:POSTMORT,CMP,[EVENT][,RCVY][,DCF]; To suppress the 5-minute automatic dumps, enter the input command beginning as follows: RLS:POSTMORT,CMP=a; For details of both messages, refer to AT&T 235-600-700, Input Messages Manual. Note that the ``RLS'' message ``unlocks'' the area where postmortem area is stored, that is, allows it to be overwritten with information about subsequent postmortems. Otherwise, the system preserves postmortem information for an hour. There may be additional error reports and status dumps that provide more information about the error(s). They will have the same event numbers as the postmortem dumps. Some information is stored in a log file on disk. To display information from the log file, enter the following message: OP:LOG:LG="DAYLOG"[:[DATE=b[&&c],][TIME=d[&&c],][KW="f",][ID=g,] [MSGCLS=l|TYPE=h,][LIMIT=i,][,CLASS=j|,DEVICE="k"]]; For details, refer to AT&T 235-600-700, Input Messages Manual. Try to analyze the cause of the initialization and verify that the CMP recovers properly. Software in the various peripheral controllers of the communication module (CM) produces several types of postmortem dumps. Each is identified by the controller which produced it. All have the following form: REPT:POST MORTEM a EVENTNO=b The peripheral controllers which produce postmortem dumps for both CM1 and CM2 are as follows: o CLNK = Communication Link o MSCU= Message Switch Control Unit o MMP = Module Message Processor o PPC = Peripheral Pump Controller o FPC = Foundation Peripheral Controller o TMS = Time Multiplexed Switch o CMP = Communication Module Processor (for software releases 5E6 and later). The basic format of the postmortem dumps on the units listed previously is as follows: REPT:POST MORTEM a EVENTNO=b cccccccc The variable field definitions are as follows: o a = hardware identity (for example, MSCU=0) o b = decimal number indicating the event number o c = hexadecimal array of eight digits in a sequence of eight words and in four lines. Each of the hardware identities listed previously is explained in detail in AT&T 235-600-750, Output Messages Manual. The postmortem dump report for the TMS hardware identity is an exception to the general format that the other hardware identities follow. The format for TMS is as follows: REPT POST MORTEM DUMP TMS=a EVENTNO=b c c c c c c c c The variable field ``cccccccc'' can appear more than once; however, the most useful information is contained in the first four words. With the various postmortem dump reports, an autonomous dump on the Network Clock is possible. The layout of this report is as follows: DUMP NC a b c NETWORK CLOCK a CCB/CLRT REGISTERS X' dddddddd This message can be used to identify the exact problem according to the bits that are set as shown in the format description. For details of this message, refer to AT&T 235-600-750, Output Messages Manual. Note: Information concerning the register layouts for the registers referred to in the error reports (in the log files) can be obtained from the Appendixes section of AT&T 235-600-750, Output Messages Manual. Two types of interrupts may occur in the AM. These interrupts are indicated by either a REPT CU a ERROR INTERRUPT or a REPT CU a MAINTENANCE INTERRUPT report. When either of these reports are printed on the ROP, more information about the interrupt is written in a log file. The AM uses three error log files: memory history log (MEMLOG), error interrupt handler log (ERLOG), and automatic postmortem log (PMLOG). Entries in the MEMLOG file and the ERLOG file are generated with an REPT CU interrupt report. Entries in the PMLOG file are generated following a system initialization and are accompanied by an REPT CU interrupt report. Each log file entry has a sequence number associated with it. Since all three log files use the same sequence counter, the order in which a set of entries occurs can be determined from the sequence numbers. These numbers appear in the REPT CU interrupt report. Each entry has a date and time stamp to relate log entries to other printouts. Therefore, it is important to save the ROP output of the last 12 hours prior to a REPT CU interrupt report to be able to extract any data that might be useful for error analysis. Memory Error Types: When a MEMLOG report must be analyzed, the type of memory error can be indicated in the report. Descriptions of these errors are as follows. o ERROR A: An OR of internal memory hardware checks resulted in error detection. If this occurs in the active control unit, a stop-and-switch occurs. At least one bit is set in error register 1 (ER1) bits 0-11 or 22. In the trapped address register (TAR), the bits 28 and 29 are both reset. o ERROR B: An out-of-range address is detected. The bits 0-11 and 22 in ER1, plus the bits 28 and 29 of TAR are all reset. o ERROR C: The Hamming check circuitry detected a multibit uncorrectable error during a system access of memory. TAR bit 29 is set. o ERROR D: A correctable data parity of Hamming check error or uncorrectable refresh error is detected (during system access or refresh access). TAR bit 28 is set. Error Interrupts: Normally error interrupts are nonfatal errors, detected by the system in either the on-line or standby control unit. However, they can escalate to a processor switch or an initialization if preset thresholds are exceeded. When an error interrupt occurs, the information printed on the ROP or contained in the log files can be used for error analysis. The log file information should be saved in case the next level of technical support is needed. The error interrupts can be classified into the following three areas: 1. Less serious hardware errors (for example, memory errors, input/output errors, or refresh parity) 2. Errors related to standby CU in a duplex (active/standby) configuration 3. Software-related errors. When error interrupts occur, the associated information is written in one of the following error log files: o MEMLOG o ERLOG. If the REPT CU a ERROR INTERRUPT b c is output, the ``a'' indicates the control unit side 0 or 1; the ``b'' indicates the interrupt type; and the ``c'' indicates the sequence number under which supplementary data is available in the particular log file. Therefore, when log file output is requested, this sequence number should be specified for the parameter ``KW'' in the input command: OP:LOG:LG=a[:DATA,DATE=b[&&c]] [,TIME=d[&&e]] [,KW=f] [,ID=g] [,TYPE=h] [,LIMT=i] [,CLASS=j|,DEVICE=k]]; The variable ``a'' equals the log file name (that is, MEMLOG or ERLOG). Maintenance Interrupts: Maintenance interrupts are output after a system initialization. These reports indicate that a maintenance reset function (MRF) has occurred. The postmortem dump, normally printed automatically or requested via OP:LOG: a message, can be used to determine the problem. The variable ``a'' in this situation equals PMLOG. See AT&T 235-600-750, Output Messages Manual, for details of the OP:LOG: a message. The postmortem dump is used to determine the reason for a particular initialization. The data provided consists of most of the critical hardware registers from both control units and some nonhardware type of information dealing with the past and present configuration of the AM. The start of an initialization is usually indicated by the following reports: o REPT CU a MAINTENANCE INTERRUPT (where a = the member number) o REPT PHASE IN PROGRESS o START OF CU a RECOVERY (where a = the member number). Normally, the first report printed is the START OF CU a RECOVERY report. The MAINTENANCE INTERRUPT indicates the associated log entry in the PMLOG. The AM postmortem dump is subdivided into the following sections: Note: The AM postmortem dump sections are described in AT&T 235-600-750, Output Messages Manual. o Heading: The PMLOG reports are either labeled MAINTENANCE INTERRUPT or POSTMORTEM DUMP. The header POSTMORTEM DUMP will appear in a manually requested initialization and sometimes when the initialization was started due to the application software. o Initialization message: This section indicates the source of initialization, the on-line processor at the time of initialization, the processor actually involved in the initialization, and the recovery action that took place. Also, the source of the request is indicated (by the SOURCE field): hardware, software, or manually requested. Note, this source does not indicate the problem source. o Requested initialization parameters. o EAI buffer. o Timer. o General registers. o Faulty CU registers. o Interrupt stack saved state. o Off-line registers. o Main store registers. o Real-time clock. The faulty control unit registers, timers, and the main store registers are primarily of interest when the initialization is a hardware request. When the initialization is software requested, the requested initialization parameters and the general registers are of interest. The initialization message should be analyzed for both types of initialization. Postmortem Analysis: When a postmortem dump is generated, the response would be a REPT CU a MAINTENANCE INTERRUPT b report, where ``b'' indicates the sequence number belonging to the postmortem dump entry in the PMLOG file. When not printed, the postmortem dump can be requested via the OP:LOG:LG="PMLOG",KW=a; (a = the sequence number). Refer to AT&T 235-600-700, Input Messages Manual, for details of the OP:LOG message. The Operating Service Position System (OSPS) maintenance is provided by AM software, switching module processor (SMP) software, and firmware located in the link adapter unit (LAU), the video display terminal (VDT), the basic services terminal (BST), and the combined services terminal (CST). The OSPS maintenance, through software in the areas of system initialization (SI) and recovery, terminal maintenance (TM), and the human/machine interface, support OSPS maintenance in the following areas: o OSPS operator positions (VDTs, BSTs, and CSTs) o Data links [Directory Assistance System/Computer (DAS/C), Real-Time Rating System (RTRS), etc.] o Remote alarms o Miscellaneous OSPS features [automatic charge quotation (AUTOQUOTE), busy line verify (BLV), etc.]. Refer to the following manuals when performing OSPS maintenance: o AT&T 250-505-100, OSPS Description and Procedures o AT&T 250-505-11X, OSPS Maintenance and Growth (X = manual number associated to the applicable software release) o AT&T 250-520-100, OSPS Directory Assistance/Listing Services Basic Services Terminal, Description and Operation o AT&T 250-520-105, OSPS Toll and Assistance Video Display Terminal, Description and Operation o AT&T 250-520-110, OSPS Combined Services Terminal, Description and Operation. Refer to AT&T 235-001-001, Documentation Description and Ordering Guide, for the complete list of OSPS manuals. The 235-XXX-XXX manuals do not provide maintenance procedures for the repair of equipment manufactured by vendors other than AT&T (for example, tape drives and disk drives). To identify the appropriate maintenance manual for each unit of such vendor equipment, refer to Section 3 of AT&T 235-001-001, Documentation Description and Ordering Guide. The objective of this section is to provide a working knowledge of LU architecture, necessary to understand and resolve complex LU problems. Note: This subsection contains only the introductory information concerning LU problems. Refer to AT&T 235-105-220, Corrective Maintenance Procedures, for the procedures needed to resolve the LU failures. Currently there are three types of 5ESS switch analog LU in use. These LUs are referred to as the Model 1 LU, the Model 2 LU, and the Model 3 LU. The Model 1 LU is a 2-shelf unit and, when fully equipped, it contains 48 circuit packs and two (-48 V to 5 V) DC power converters. The Model 1 LU grids contain three circuit packs each. The Model 2 LU is also a 2-shelf unit and, when fully equipped, contains 38 circuit packs and two (-48 V to 5 V) DC power converters. The Model 2 LU grids contain two circuit packs each. The Model 3 LU is a 2-shelf unit. When Model 3 is fully equipped, it contains 40 circuit packs and two DC power converters. The Model 3 LU consists of 10 grids. Each grid consists of two circuit packs. The Model 1, Model 2, and Model 3 LUs are fused identically. At the power distribution bay, one 20-amp fuse for each LU is used to provide -48 V DC power to the SM cabinet local fuse panel. Twelve 70-type fuses, at the local fuse panel, are assigned to distribute -48 V to each LU. The LU is a peripheral unit and is located in the SM. Up to eight LUs may be assigned to one SM. The most important function that an LU has to perform is to connect an analog subscriber line to the 5ESS switch. To accomplish this, the line must be connected to a channel. The analog voice then is converted to pulse code modulation (PCM) digital data, and the PCM digital data is then put on a peripheral time slot. One fully equipped LU: o Model 1 or Model 2: Has the capability of serving 512 subscribers o Model 3: Has the capability of serving 640 subscribers. With 64 peripheral time slots available to each LU: o Model 1 or Model 2: Has a ratio of 512 lines to 64 time slots, or a concentration ratio of 8:1. o Model 3: Has a ratio of 640 lines to 64 time slots, or a concentration ratio of 10:1. Any condition that causes an LU service group to be removed from service, also causes the line to channel concentration ratio of that LU to be changed (for example, in a Model 2 LU, it changes from 8:1 to 16:1). This will usually result in slow dial tone, no dial tone, or incoming calls going to recorder. To avoid customer complaints, it is recommended that LU service groups not be removed from service during heavy traffic hours. If a service group is forced OOS automatically, by peripheral fault recovery, it should be treated as a service-affecting condition and repaired and restored to service as soon as possible. Each LU is controlled by a peripheral interface control bus (PICB). Also, each LU is connected to a peripheral interface data bus (PIDB). The PIDB is used to send and receive PCM digital voice time slots, between the LU and the time slot interchanger (TSI). Each LU is assigned a total of 64 peripheral voice time slots, 32 time slots for each service group. There are also 32 channels in each LU service group. The 64 LU channels are dedicated to the 64 PIDB time slots. Each LU grid services 64 subscribers. When a grid in the Model 1 LU is removed from service, 64 subscribers are removed from service. The grid in Model 2 and Model 3 LUs can be removed from service at the half-grid level. When a Model 2 or Model 3 LU half grid is removed from service, 32 subscribers are removed from service. Any LU grid OOS condition is service affecting and should be resolved as soon as possible. See AT&T 235-105-220, Corrective Maintenance Procedures, for details on restoring OOS grids to service. The LU A-LINKs are wired paths between the first and second stage switches in LU grids. If an A-LINK is OOS, a path is not available through the concentrator grid network. An accumulation of OOS A-LINKs can cause network blockage and can be service affecting. To avoid subscriber complaints, OOS A- LINKs should be restored to service as soon as possible. See AT&T 235-105-220, Corrective Maintenance Procedures, for details on restoring OOS A-LINKs to service. The LU hardware is not duplicated. If an LU function is out of service, that function is unavailable for call processing. If operational test failures (OTFs) are occurring or REX grid fabric exerciser failures are being reported, the affected LU is being forced to complete calls with defective hardware. This can be service affecting and a source of subscriber complaints. A grid fabric exerciser fault in the grid of LU Model 1 or halfgrid of LU Models 2 or 3 changes its MCC display state to degraded (DGR). In summary, restore OOS hardware to service as soon as possible. Take action to resolve operational test failures and grid fabric exercise faults. Refer to AT&T 235-105-220, Corrective Maintenance Procedures, for assistance in resolving LU faults. Any LU fault that cannot be resolved at the local level should be supported by the next level of technical support. The terminal maintenance subsystem is designed to detect faults in lines, trunks, and associated equipment. Fault detection is performed either automatically or manually by software controlled tests. Testing is performed locally by utilizing the TLWS capabilities. Remote testing can be implemented from centralized test systems such as the following: o Local Test Desk (LTD) o Mechanized Loop Test (MLT) o Remote maintenance center, for example, SCCS o Centralized Automatic Reporting on Trunks (CAROT) System o Centralized Trunk Test Unit (CTTU). These remote test positions have powerful testing capabilities to supplement the local TLWS. Per-call testing by call processing software is a primary means of line fault detection. A number of these tests are performed on every call processed by the 5ESS switch. Per-call tests are performed independently on both the originating and terminating sides of the call. In addition, tests are done at one of three phases of a normal call. These are as follows: a. Origination: Origination is the interval between the calling party off-hook detection and talking path connection. b. Termination: Termination is the interval from when the called party line is determined to be idle to when one of the following occurs: o Calling customer goes on-hook o A talking path connection is broken down. c. Disconnect: Disconnect is the interval between customer on-hook and line restoration to idle. When a per-call test detects a failure, all data associated with the call is sent to terminal maintenance for a test failure that indicates a trouble on the line. Trouble indications within the line unit are referred to switch maintenance. The problem is then analyzed, and a decision is made on what course of action is to be taken. This could result in any of several maintenance actions such as diagnostic tests, changing equipment status states, or a system alarm. Routine tests are periodic maintenance checks run by the terminal maintenance subsystem. These tests are used to assure trunk and line integrity. Routine tests are run on circuits that are assumed to be in good operating condition. These tests can be initiated either automatically or manually. Flexible scheduling of automatic trunk testing is possible through automatic progression testing (APT). In APT, a test history keeps track of information concerning the tests. This allows interruptions of the testing cycle when the trunks are needed for service. When traffic subsides, the testing resumes where it left off. Test results are analyzed and displayed locally and/or at a remote testing location. Routine trunk tests can be classified as operational or transmission tests. Operational testing of trunks encompasses the following objectives: o Verify the operational characteristics of interface and carrier facilities and distant central office equipment. o Verify the end-to-end ability to detect and send signaling and supervision. o Routinely exercise the interface circuits in a distant central office. A trunk error analysis (TERA) analyzes MDIIs that result from trunk or universal tone decoder failures. The results (pass or fail) of trunk operational tests are also routed to TERA. When TERA determines that a trunk or universal tone decoder is experiencing a high-error rate, recovery actions are initiated. The recovery actions can consist of an output message, or, when applicable, an operational test on an outgoing trunk. Refer to AT&T 235-105-119, Maintenance Guide Utilizing OMS5, for details of how to use TERA. For information about the activation of TERA, refer to the appropriate RC manual (AT&T 235-118-XXX, where XXX = manual number associated to the applicable software release. See AT&T 235-000-000, Numerical Index -- Division 235 and Associated Documents, for specific manual numbers) that contains the RC symbolic view name RC_TERA. The 5ESS switch supports incoming test calls for operational tests. The operational test for lines is the automatic line insulation test (ALIT). The ALIT is an automatic test system that scans lines and identifies faults before they affect normal service. Many tests and functions are provided to aid in trunk and line testing. These include the standard test line (TL) and functions used in previous switching systems. The routine test facilities provided include the following: o 100TL (Balance) o 101TL (Communication) o 102TL (Milliwatt) o 103TL (Signal supervisory) o 104TL (Transmission measuring and noise checking) o 105TL (Automatic transmission test line) o Synchronous test line o Nonsynchronous test line o Permanent-busy test line o Touch-tone dialing test line o Open circuit test line. The per-call tests (lines only) are as follows: a. Origination of calling party: o False cross and ground test o Power cross o Continuity check. b. Termination of called party: o False cross and ground test o Power cross o 20-Hz ringing current o Loop resistance to detect pretrip o Continuity check. c. Disconnect: o Restore verify. The Call Monitor provides an early detection mechanism for loss of call processing functionality when all other system indicators appear normal. The Call Monitor reports to the craft by ROP and an alarm indicator on MISC Page 116 when a failure in call completion analysis occurs. The ROP output is in the form of a REPT CALLMON 5- or 15-minute report. The ROP output message has either a major, minor, or no alarm. The failure criteria is defined as follows: o For the 5-minute report, failure occurs if more than 50 percent of the total calls attempted in a 5-minute period are not passed. o For the 15-minute report, failure occurs if more than 90 percent of the total calls attempted in a 15-minute period are not passed. The major alarm criteria is defined as follows: o For the 5-minute report, a major alarm occurs if 40 percent or more of the total tests are ``operational test failures.'' o For the 15-minute report, a major alarm occurs if 50 percent or more of the total tests are ``operational test failures.'' The minor alarm criteria is defined as follows: o For both the 5- and 15-minute reports, a minor alarm occurs if 70 percent or more of the total tests are ``indeterminate'' plus ``not attempted'' failures. If no alarm criteria is met, no alarm will be printed with either analysis report. The Call Monitor will perform separate analyses for common channel signaling (CCS) test calls (if equipped) along with non-CCS test calls. It utilizes the Terminal Maintenance APT functionality to make these operational test calls. Non-CCS test calls are based on the default APT test for the trunk group in the AM ODD. All CCS test calls use the Voice Path Assurance (VPA) continuity test. For 5E9(1) and later, ability to inhibit the Call Monitor on a per-trunk-group basis is provided by a new field in the static AM ODD relation RT_TRKG. The new field, ``callmon_inh'', is populated from recent change trunk view 5.1 (as is the existing field for inhibiting APT). If APT is inhibited, then the Call Monitor must be inhibited. The monitor routinely cycles through the AM ODD for trunk groups and selects trunk groups to use for making the test calls. A test call will be attempted every 30 seconds. The monitor can be inhibited as well as requested to print the past 15-minute history and print per-test call results (verbose mode). The alarm indicator on MISC Page 116 can also be retired. Additional information on the Call Monitor is provided in Sections .RM 3./ and .RM 4.2/ of this document and in AT&T 235-105-210, Routine Operations and Maintenance Procedures. Recent change (RC) is a system function that allows maintenance personnel access to the 5ESS switch data base. Recent change is used to add to or delete from the data base. Also, recent change is used to update or verify the data base using a select/mask format. The data base for the 5ESS switch supports a relational data base with the following methods of access: o Recent change/verify (RC/V) o Office data administration (ODA) o Office record programs (called views because they are user-oriented) o Remote memory administration system (RMAS). For details concerning the recent change and verify subsystem, refer to Section .RM 3.10/ of this manual. Field update is the process of activating orderly program changes in the 5ESS switch software. In-service offices receive most official software changes in the form of software updates. The software update originates as a program enhancement or as a fix for a problem within the software release. Function, file, and byte replacement are the three types of software updates provided. The 5ESS switch software updates usually replace entire sections of program software as compared to the word-by- word changes of other Electronic Switching Systems. Aiding in the updating of a 5ESS switch are three external interfaces to the program update subsystem. These three interfaces provide for the generation, distribution, and activation of software updates. The interfaces are as follows: o A Programmer Support System (PSS) o A SCANS o Remote maintenance center (for example, SCCS). The PSS originates program updates via the generation and initial distribution of software updates. After a software update has been composed, tested, and approved at the PSS, it is assigned a software update identification number and transmitted to SCANS. In an emergency (such as SCANS outage), a software update could be transmitted from the PSS over a data link directly to the office if the situation requires immediate action to maintain switching system integrity. Craft and/or maintenance personnel at the remote SCC or 5ESS switch would have to make a verbal request to the program update coordinator at the PSS. The coordinator would set up the emergency data link (from the PSS to the 5ESS switch) and manually transmit the software update, after maintenance personnel primes the 5ESS switch for reception of the software update files. The SCANS is an AT&T time-shared computer system for distributing software updates. It is usually accessed by maintenance personnel at the remote SCC work station using the No. 2 remote SCC computer to receive and record the software updates. The SCANS can also be accessed by maintenance personnel at the local office. The SCANS should be checked prior to inserting or activating any software updates to ensure that they have not been canceled or changed. The remote SCC provides for remote activation of software updates at a 5ESS switch. It uses a 1200-baud dial-up terminal to access SCANS and activates the program update subsystem to apply the software update. Communication between the remote SCC and the program update subsystem is via the maintenance channel. It is not necessary to have maintenance personnel present at the local office to aid in the activation of the software update(s). Although software updates are usually activated by the remote switching control center, they may also be activated locally through the MCC video terminal for one or more of the following reasons: a. The remote SCC option is not carried by the 5ESS switch being accessed. b. The remote SCC (if the option is carried) is down, and a software update must be activated to maintain switching integrity. Note: Reasons (a) and (b) require that a 1200-baud terminal be present at the 5ESS switch. The terminal must be full duplex, be capable of printing at least 80 characters per line, and must have a 212A-type data set. This terminal is used to access SCANS and poll the SCANS data base for relevant software updates. c. Local control of the software update activation is desired. This carries the following two options: 1. The entire activation procedure (including access of SCANS) is to be done locally. 2. The remote SCC accesses SCANS and turns control of the activation over to local personnel at the MCC. The software update activation responsibility between AT&T and a switch owner (normally an OTC) is as follows: a. During preturnover (new office), retrofit, and restart intervals, the AT&T installer is responsible for obtaining and activating all current software updates in the SCANS data base which apply to that office. b. At all other times, in a working office (when not in a retrofit or restart mode), the switch owner or the remote switching control center is responsible for obtaining and implementing all applicable software updates. Regular field updates are done in a timely and orderly fashion through software updates. Unexpected problems with the software release can occur that require immediate correction, not allowing time for the normal software update development and issue processes. Emergency fixes are accomplished on a word- by-word basis under direction of the AT&T Customer Technical Support (CTS) Organization. Emergency fixes are assigned a sequential craft number similar to the software update number. The program update subsystem provides emergency fixes with the same status and options as software updates (that is, make temporary, make permanent, backout). Emergency fixes specify the address to be changed, the new data to be inserted, and the old data to be matched. Emergency fixes are also known as address-data couplets. As with software updates, most emergency fixes are activated remotely from the remote SCC. Communication between the remote and local program update subsystem is via the maintenance channel. It is not necessary to have maintenance personnel present at the local office to aid in the activation of the fix. Emergency fixes may also be activated locally through the MCC if the following occurs: a. The 5ESS switch does not carry the remote SCC option. b. The remote SCC (if the option is carried) is down, and the fix must be activated to maintain switching system integrity. c. Local control of the fix is desired. Software release retrofit refers to the software and procedures used to replace the resident software release text and data bases [ECD, ODD, and System Generation (SG)] in an operational 5ESS switch with new software release text and evolved data bases. Software release update refers to the software and procedures used to replace the resident software release text in an operational 5ESS switch. Software release update allows for replacement of text for installation of a software update (formerly BWM) consolidation load or a software point load release. Software release update does not include evolution and replacement of the SG or ODD data bases. Although there is no ECD evolution, the application of point load specific keystroke files to the ECD is allowed. Large terminal growth (LTG) refers to the software and procedures used to add large quantities of lines and trunks to an operational 5ESS switch by evolution and replacement of the ODD data base. It does not include replacement of the software release text, ECD, or SG data bases. Software release retrofit, software release update, and LTG are referred to collectively as software release transitions. New software release text and data bases are delivered to the office on magnetic tapes supplied by AT&T. Recent change activity should be kept to a minimum or stopped, if possible, for 2 weeks prior to a software release retrofit or LTG. Prior to all transitions, a copy of the existing software release should be saved on spare disk packs or magnetic tapes. In some cases, associated hardware and/or read-only memory (ROM) circuit pack changes may be required prior to starting the transition process. When the transition process begins, however, all essential duplex and simplex equipment must be operational. Although stable 2-port circuit-switched calls are saved over a software release transition initialization, a software release transition should be planned for a low-traffic period to minimize the number of calls that might be affected by call processing interruptions. The types of calls not saved during a software release transition initialization include calls involving more than two ports, such as calls using conference circuits. Transient calls (that is, originating, dialing, ringing, calls on hold, calls being forwarded/transferred, etc.), packet calls, and test calls are also not saved. For detailed information concerning software release retrofit procedures, refer to AT&T 235-105-24X (X = manual number associated with applicable software release). For detailed information concerning software release update procedures, refer to AT&T 235-105-34X (X = manual number associated with applicable software release). For detailed information concerning LTG procedures, refer to AT&T 235-105-44X (X = manual number associated with applicable software release). For additional information, refer to AT&T 235-000-000, Numerical Index - Division 235 and Associated Documents. Note: The Design Change Management System (DCMS) data base is not restricted to only covering the CNs for the 5ESS switch. This data base provides CN coverage for all of the AT&T Network Systems products. The DCMS data base is the official vehicle for notification of product changes (that is, CNs) for all of the AT&T Network System products. Also, this data base notifies the customer of service enhancements that can be purchased. The DCMS data base service is free to the customers. The following information concerning CNs is provided to the customer by DCMS: 1. DOCUMENTATION on the changes that are made to AT&T Network Systems products. Documentation is in the form of a product change notice (PCN). A PCN is issued for each change. The PCN document consists of the following: a. Product being changed b. Reason for the change c. Description of the change d. Effect that the change will have on the customer's operations during implementation e. Affected product drawings and their titles f. A comparison of the markings that the product will bear before and after the change g. Other miscellaneous information. 2. APPLICATION STATUS REPORTS that track and report on the status of the implementation of these changes in the offices that employ affected products. Application status reports are compiled interactively. The reports illustrate the current status of the offices that are affected by product changes. The status reports can be obtained in a standard format or can be customized on an ad hoc basis to meet the DCMS user's specific needs. 3. HISTORICAL INFORMATION by CN, office, or product once the implementation process has been completed. Historical information is available on an ad hoc basis. The DCMS data base covers CN office implementations made by an AT&T installation group from 1983 to the present date. The DCMS user's manual explains how to use the menu driven DCMS data base. The manual is distributed by the National CN Engineering Group located at the AT&T Southwestern Region. With a proper ID number and login, customers can obtain an on-line version of the manual. For additional information, call the telephone number shown in the address for Southwestern Region as follows: AT&T Southwestern Region Department NF93D6T00 1111 Woods Mill Road Ballwin, Missouri 63011 (314) 891-2930 The DAP terminal can be used to perform all commands/functions that are needed to maintain the switch. A maximum of 8 DAP terminals for 5E6 or 16 for 5E7 and later can be accessed. These terminals are defined in the data base. The master control center (MCC), trunk and line work station (TLWS) (TLWS is mode of MCC), supplementary TLWS [with the exception of being able to access the emergency action interface (EAI) page display] are DAP-type terminals. Non-DAP terminals can be used to perform the same functions that the DAP terminals perform with the exception of being able to access the MCC page displays. When using a non-DAP terminal, input messages (message function mode) must be entered to maintain the switch. No input commands (for example, poke commands) can be entered from a non-DAP terminal. The MCC uses four function keys. When one of these keys (Figure .AW G9/) is depressed, the system performs the corresponding function. The keys are as follows: o ALM RLS: Alarm release o CMD/MSG: Input command or input message o NORM DISP: Normal display o EA DISP: Emergency action display [can only be performed from the MCC or switching control center (SCC)]. When the system is in the manual retire mode, the ALM RLS function key releases audible and visual alarm indications (CRITICAL, MAJOR, or MINOR, and flashing indicators). Flashing indicators go to steady reverse video when retired. Note: The minor alarm audible is self-releasing after 5 seconds, but its visual indication must be released manually. The command/message (CMD/MSG) function key configures the MCC to accept either input CMDs or input MSGs. The key acts as a toggle and allows input in one mode or the other. The user may switch between either mode after acknowledgment of the previously entered message. Any unexecuted data in either area is lost if a switch is made before an acknowledgment is received. Unexecuted data in the input message area is normally saved until an intercharacter time-out occurs. If a time-out occurs before the message is completed, all data is lost. The position of the cursor on the video display indicates which input mode the MCC is in. The cursor resides in the input message line area while in the MSG mode. If the MCC is in the CMD mode, the cursor resides at the CMD entry area (at the top left of the control and display area). Whenever the display is brought on-line or a new page is selected, the input mode will remain unchanged. The NORM DISP function key places a page controlled from the administrative module (AM) on the display. The page displayed will be the previously displayed page. Depressing the NORM DISP key again will redraw a clean display without aborting any processes in progress. The EA DISP function key enables emergency action mode and displays the EAI page on the screen. This page is used during a system emergency for system recovery functions. Depressing the EA DISP function key again will abort all incompletely entered actions and redraw the emergency action interface page display. Note: The emergency action mode can only be enabled from the MCC or SCC. Most other keyboard keys are used in a normal fashion to enter numeric codes, input messages, and alphanumeric responses to the system. Their input is respective to the symbol designated on the key. Certain keys are used for line administration as explained in the remainder of this section. The following procedure may be used to access an MCC page display: 1. Select a normal system display page by depressing the NORM DISP function key. 2. If the maintenance terminal is not in the command mode, depress the CMD MSG function key. Note: When in the command mode, the CMD:_ prompt is displayed and the cursor is positioned in the command input area in the upper left-hand portion of the display. 3. Enter the desired MCC page number (for example, 100 - Page Index display) that is to be displayed. A capability is provided to specify in their equipment configuration data base (ECD) getty forms the page that is to be displayed automatically on each display device following a terminal restore (for example, initialization, system boot, or remove/restore). The process that initializes a page specified in the ECD must be connected to a port. Therefore, not every page in the system can be displayed automatically following a terminal restore. The following list contains the page names and port numbers of the system pages that may be specified in an ECD getty form: Pagename Portid CPDP 17 INDEX 17 CDUP 17 For applications not desiring a specific display upon a terminal restore, the common processor display page (CPDP) is the default page for the maintenance terminals. If another system page is requested and displayed, depressing the NORM DISP function key causes the last page requested to be displayed again. The following procedure can be used to specify the initial page for a display device. 1. At the maintenance terminal, invoke RC/V by entering 199. 2. Fill in the necessary information to initiate an update of the getty form for the desired display device. Procedures for updating data base forms are explained in the ECD/SG Data Base Manuals [for example, AT&T 235-600-306 for 5E6, AT&T 235-600-307 for 5E7, AT&T 235-600-308 for 5E8, and AT&T 235-600-309 for 5E9(1)]. 3. When the getty form is displayed, enter the name of the initial page desired in the pagename field and the port number of the process that initializes the page in the portid field. 4. After updating the getty form, remove and restore the device for which the getty form was updated. One of the following methods can be used to switch one or more switchable devices [that is, the receive-only printer (ROP) and/or maintenance terminal] to an alternate peripheral controller: a. Input message SW:PORTSW, refer to the AT&T 235-600-700 for details. Note: The port switch must be in the AUTO position. b. Poke input command (401 - PORTSW, 402 - ROP, or 403 - MCRT) illustrated on the 111/112 - AM, AM Peripherals page display. Note: The port switch must be in the AUTO position. c. Flip the port switch associated with the switchable device to either 0 or 1. (0 = peripheral controller 0, 1 = peripheral controller 1). Note: In either of these two positions, the port switch cannot be reconfigured using the previous two methods. The MCC provides the ability to select a control and display page at any time. The index display is brought into the control and display area by entering 100 (into the CMD entry area) and executing it. The 100 index page consists of a list of possible MCC control and display pages. Those pages not shown are requested by entry from oth