Module 54 - T304

T304: Applying Your Test Plan to Field Management Stations (FMS) - Part 1 Signal System Masters (SSM) Based on NTCIP 1210 Standard v01

HTML of the PowerPoint Presentation

(Note: This document has been converted from a PowerPoint presentation to 508-compliant HTML. The formatting has been adjusted for 508 compliance, but all the original text content is included, plus additional text descriptions for the images, photos and/or diagrams have been provided below.)

Slide 1:

(Extended Text Description: Welcome - Graphic image of introductory slide. A large dark blue rectangle with a wide, light grid pattern at the top half and bands of dark and lighter blue bands below. There is a white square ITS logo box with words "Standards ITS Training - Transit" in green and blue on the middle left side. The word "Welcome" in white is to the right of the logo. Under the logo box is the logo for the U.S. Department of Transportation, Office of the Assistant Secretary for Research and Technology.)

Slide 2:

(Extended Text Description: This slide, entitled "Welcome" has a photo of Ken Leonard, Director, ITS Joint Program Office, on the left hand side, with his email address, Ken.Leonard@dot.gov. A screen capture snapshot of the home webpage is found on the right hand side - for illustration only - from August 2014. Below this image is a link to the current website: www.pcb.its.dot.gov - this screen capture snapshot shows an example from the Office of the Assistant Secretary for Research and Development - Intelligent Transportation Systems Joint Program Office - ITS Professional Capacity Building Program/Advanced ITS Education. Below the main site banner, it shows the main navigation menu with the following items: About, ITS Training, Knowledge Exchange, Technology Transfer, ITS in Academics, and Media Library. Below the main navigation menu, the page shows various content of the website, including a graphic image of professionals seated in a room during a training program. A text overlay has the text Welcome to ITS Professional Capacity Building. Additional content on the page includes a box entitled What's New and a section labeled Free Training. Again, this image serves for illustration only. The current website link is: http://www.pcb.its.dot.gov.)

Slide 3:

T304: Applying Your Test Plan to Field Management Stations (FMS) - Part 1 Signal System Masters (SSM) Based on NTCIP 1210 Standard v01

(Extended Text Description: Title slide: T304: Applying Your Test Plan to the FMS-Part 1 Signal System Masters Based on NTCIP 1210 Standard v01 - The slide presents two graphic images under the title that show a testing set for SSM on the left image with two persons engaged in testing matters and associated equipment such as oscillography machine that records data and SSM which performs functions. Image on right shows a SSM cabinet opened and interconnected to a bulb matrix array for showing indications. Both images together convey the meaning of SSM being tested and this module deals with the subject.)

Slide 4:

Instructor

Raman K. Patel, Ph.D., P.E.

President

RK Patel Associates, Inc.

New York City, NY, USA

Slide 5:

Learning Objectives

• Describe within the context of the system lifecycle the role of a test plan and testing to be undertaken
• Recognize the purpose, structure, and content of well-written test documentation for an SSM based on IEEE 829-2008 formats
• Explain how to develop the complete test documentation package for an SSM specification based on NTCIP 1210 Standard v01
• Describe the testing of an SSM using a sample test document

Slide 6:

Learning Objective 1

• Describe within the context of the system lifecycle the role of a test plan and testing to be undertaken

Slide 7:

How Is an SSM Used in a Traffic Management System?

Role of a Signal System Master (SSM)

SSM is a Portion of a Field Management Station (FMS)

Source: FHWA

SSM Coordinates

Signal System Locals-SSLs

(Intersection Controllers)

(Extended Text Description: Author's relevant description: The layout depicts how an SSM is used to perform its main functions: an SSM coordinates signal timing on arterial sections by monitoring and controlling SSLs. A field network of SSM layout is shown on right side. An SSM is connected to several SSMs in three sections. SSM is performing a coordination of SSLs function.)

Slide 8:

How Is an SSM Used in a Traffic Management System?

How an SSM Is Used Within the Typical Physical Architecture

(Extended Text Description: Author's relevant description: This slide depicts the typical physical architecture used by NTCIP 1210 deployments. NTCIP 1210 deals with the link between the TMS and SSM. As shown in the slide, this standard is closely related to NTCIP 1202; that is because an SSM communicates to the local controller using NTCIP 1202. Part of the goal of NTCIP 1210 is to standardize how the SSM interfaces with the local signal controller using NTCIP 1202 v02. On the left side, a Management station and a laptop are connected to a SSM and SSL and SSL at end connects to a traffic signal head is shown. SSM connects to a SSL. Together, the layout represents a typical architecture.)

Source: NTCIP 1210, Fig. 3, p. 13

Slide 9:

Purpose of Testing an SSM

Testing Is a Process That Uses a Documented Test Plan

(Extended Text Description: Author's relevant description: Key Message: Testing is a process that uses a documented test plan designed to check conformance to the SSM standard. Testing verifies that SSM requirements are fulfilled. Was the system built right? Check if the unit exhibits the functionality. An SSM system's conformance to a documented test plan based upon an ITS standard is our main focus in this course. On left slide a SSM with open door is shown and on right side a PC set up with a test plan document is shown. A curved arrow connects both.)

Slide 10:

Purpose of Testing an SSM

Testing Methods Used for Conformance Verification

(Extended Text Description: Author's relevant description: Testing Methods Used for Conformance Verification - This slide shows multiple images to explain each type of testing method. On the left first image is of a SSM with visual observation title underneath, next to it is a demonstration of several green cabinets seating on the floor with wiring; still on left lower side an analysis method is shown with a magnifying glass pointing to a man looking at bunch of SSMs being under test. On the right side of the slide, a man named Henry is standing with his arm on a SSM cabinet with open door and a testing documentation underneath it. This image conveys testing of SSM is carried out using a pre-developed test plan.)

Slide 11:

Purpose of Testing an SSM

System Lifecycle and Testing to Be Undertaken

(Extended Text Description: System Life Cycle and Testing to Be Undertaken - Vee diagram is used to explain at what stage in life cycle we prepare SSM testing document- Documentation Preparation-specification. A bracket is shown pointing to high level, detailed design stage. On the right leg of the vee-Communications interface Level Tests to be undertaken is pointing to a red circle that covers stages of testing-beginning with unit/device test. A red curved arrow connects both legs of Vee to indicate role of testing documentation. A very detailed explanation of how a Vee diagram is used in PCB modules is provided below. This level of detail is not necessary and may be skipped. Explanation of V Diagram A graphic of the systems engineering process (SEP) is shown. The main graphic of the SEP is a V-shaped diagram with some additional horizontal "wings" on the left and right side of the top of the V. Starting from the left "wing" the steps are regional architecture, needs assessment, concept selection, project planning, and systems engineering management planning. At this point the steps begin to descend the left side of the V with concept of operations, system requirements, high-level design, detailed design, and software hardware and field installation. At this point the steps begin to ascend the right side of the V with unit device testing, subsystem verification, system verification and deployment, system validation, and operations and maintenance. At this point the steps are on the right "wing" of the V with changes and upgrades and retirement/replacement. The following arrows supplement the figure: 1. A time line at the bottom of the figure indicating that all steps proceed over time with the steps forming the V overlapping one another. 2. A downward arrow on the left side of the V that indicates that these steps deal with decomposition and definition. 3. An upward arrow on the right side of the V that indicates that these steps deal with integration. 4. A horizontal arrow near the bottom of the center of the V that connects the detailed design step to the unit testing step. 5. A horizontal arrow a little higher that connects the subsystem requirements step with the subsystem verification step. 6. A horizontal arrow a little higher that connects the system requirements step with the system verification step. 7. A horizontal arrow near the top of the V that connects the concept of operations step with the system validation step. Finally, major steps are separated by a barrier that is labeled "document approval." This graphical view of SEP life cycle process used to show where SSM user needs/requirements are located. In this slide, at the top of the Vee, there are three boxes in yellow.)

Slide 12:

Verification Methods Specific to SSM Functions

Unit/Device (Bench) Testing

• SSM under test in a lab or workshop environment
• Testing with PC-based test software
• Exercise of the SSM data elements and dialogs to check conformity with standard

Cautionary Word on Unit Testing

• Need to prioritize tests by failure consequences, amount of time (available to test), and boundary test, as opposed to trying to test over 100% span
• NTCIP 1210 v01 data elements are exercised for key functionality to the extent possible

(Extended Text Description: Author's relevant description: Unit/Device (Bench) Testing On the upper right corner, a unit test SSM set up is shown with a man adjusting controls on front panel of the device under test.)

Source: ITE OET DMS-Patel

Slide 13:

Verification Methods Specific to SSM Functions

Subsystem Verification (Is the system being "built right"?)

SSM requirements will be tested to ensure that the SSM communicates with SSLs properly, including use of central software.

Examples

3.3.1 Support Basic Communications
Requirements for making requests follow.

3.3.1.1 Accept Data from the TMS
The SSM shall accept data (e.g., configuration data, commands, etc.) from the TMS.

3.3.1.2 Deliver Data to the TMS
The SSM shall deliver data (e.g., configuration data, status, etc.) to the TMS. If not specified, the response start time shall be not greater than 2000 milliseconds.

3.3.1.3 Explore SSM Data by the TMS
The SSM shall allow the TMS to discover what data and data instances are supported by the SSM. If not specified, the response start time shall be not greater than 2000 milliseconds.

3.3.1.4 Accept Data from the SSLs
The SSM shall accept data (e.g., configuration data, status, etc.) from the SSLs.

3.3.1.5 Deliver Data to the SSLs
The SSM shall deliver data (e.g., configuration data, commands, etc.) to the SSLs.

(Extended Text Description: Author's relevant description: Subsystem Verification - A SSM is connected to a vertical layout of four SSLs is shown right side. A signal head is connected one of the SSL.)

Slide 14:

Verification Methods Specific to SSM Functions

(Extended Text Description: Author's relevant description: System Verification Ensures That the Entire System Meets-System Requirements—the Physical Architecture. Key Message: End to End Testing – the entire physical architecture is shown. A central TMS system is likely to communicate with multiple devices (e.g., SSMs, SSLs – traffic controllers, ramp meters, dynamic messages signs, video detection cameras, etc.). Each device needs to be tested, including the NTCIP interface to that device. In this module, it is the SSM that we are concerned with. Each of the SSM devices can be thought of as part of a subsystem of the larger TMS system. All of these subsystems are working at once. We should be able to monitor the video detection cameras on the freeway at the same time we are operating the SSM-based signal system and the dynamic message signs and ramp meters.)

Slide 15:

Verification Methods Specific to SSM Functions

System Validation Shows Whether the "Right" System Is Built

Implemented system is validated against specified user needs to support system operators, including communications

(Extended Text Description: Author's relevant description: System Validation Shows Whether the "Right" System Is Built - Same layout shown in slide 8 is used in this slide to convey that validation testing is done for the entire SSM architecture. Key Message: Validation comes at the end: Does the SSM perform as intended? A central TMS system is likely to communicate with multiple devices (e.g., SSMs, SSLs – traffic controllers, ramp meters, dynamic messages signs, video detection cameras, etc.). Each device needs to be tested, including the NTCIP interface to that device. In this module, it is the SSM that we are concerned with. Each of the SSM devices can be thought of as part of a subsystem of the larger TMS system.)

Slide 16:

Slide 17:

Question

Which is NOT part of the testing process in a system lifecycle?

Answer Choices

1. Test planning
2. Preparation of test documentation
3. Test execution and reporting
4. Identification of system requirements

Slide 18:

Review of Answers

a) Test planning
Incorrect. Test planning is done when system requirements have begun.

b) Preparation of test documentation
Incorrect. Test documents are created during high-level design and detailed design.

c) Test execution and reporting
Incorrect. Test execution and reporting are done at each level of the testing workflow using test documentation.

d) Identification of system requirements
Correct! Identification of system requirements is NOT a part of the testing process.

Slide 19:

Learning Objectives

• Describe within the context of the system lifecycle the role of a test plan and testing to be undertaken
• Recognize the purpose, structure, and content of well-written test documentation for an SSM based on IEEE 829-2008 formats

Slide 20:

Learning Objective 2

• Recognize the purpose, structure, and content of well-written test documentation based on IEEE 829-2008 formats

Slide 21:

Purpose of Test Documentation in an SSM Specification

Objectives of the SSM Testing Documentation

(Extended Text Description: Author's relevant description: Objectives of the SSM Testing Documentation - A cover page of IEEE 829 standard document is shown at the right side. A text box is shown at lower left side with the text: "SSM Communications Interface Specification Testing Documentation Testing Documentation is made part of the SSM Communications Interface Specification." Additional bullet items point to the cover to the right:

Objectives of the SSM Testing Documentation

1. Outline What to Test
2. State Clearly How to Test
3. Report Results/Outcomes During/After Test
4. Use IEEE 829-2008 Formats

)

Slide 22:

What Is a Test Plan?

From IEEE 829-2008 Standard

• Test Plan is a document describing:
  • Scope (technical management)
  • Approach
  • Resources needed
  • Schedule to complete
• Test Plan identifies
  • Test items
  • Features to be tested
  • Testing tasks
  • Risks requiring contingency plan

Slide 23:

Structure of a Test Plan

From IEEE 829-2008 Standard

Sets overall workflow context

(Extended Text Description: From IEEE 829-2008 Standard A text box set shows MTP at top and three boxes below MTP shows three types of LTPs (Unit Test Plan, Subsystem Integration Test Plan, System Acceptance Test).

The MTP points to the following bullet items:

• Integrity level scheme and choice
• Overall test processes, activities, and tasks
• Test levels and documents

The three types of LTPs point to the following bullet items:

• Scope of test level
• Resources
• Test method(s)

)

A Master Test Plan may not always be required!

Slide 24:

Structure of a Test Plan

MTP Structure Provides for Workflow for Multiple Devices

(Extended Text Description: Author's relevant description: MTP Structure Provides for Workflow for Multiple Devices - Slide has two levels for testing, one for SSM and one for SSLs. Each level has text boxes. The top level has SSM Unit Test Plan, SSM Subsystem Integration Test Plan, SSM Acceptance Test Plan. The bottom level has SSL Unit Test, SSL Subsystem Integration Test Plan, SSL Subsystem Integration Test Plan.)

Slide 25:

Structure of a Test Plan

SSM Test Plan Structure Based on IEEE 829-2008

(Extended Text Description: Author's relevant description: SSM Test Plan Structure Based on IEEE 829-2008 - A vertical text box layout is shown on the left side: At top Test Plan box is connected with an arrow to Test Design text box, which is connected to Test Case box. Test Case box is connected with two headed arrow to Test procedure box at end. To the right is the following text corresponding to the four box layout:

Test Plan describes the Overall Approach to SSM Testing.

Test Design specifies the details of the test approach - what is to be tested. It is shown here for Unit Test - similar designs exist for Integration Test and Acceptance Test.

Test Case specification outlines a set of test inputs, execution conditions, and expected results (outputs).

Test Procedure specification defines the steps to execute a test.)

Slide 26:

Content of a Test Plan

Level Test Plan (LTP) Outline per IEEE 829-2008

• Introduction
  • Document identifier, scope, and references
  • Level in the overall sequence (First Unit Test...)
  • Test classes and test conditions
• Details of the Level Test Plan
  • Test items and their identifiers
  • Protocol Requirements List (PRL) for NTCIP 1210 objects and dialogs (may include RTCTM)
  • Features to be tested/not to be tested
  • Test approach
  • Pass/fail criteria
  • Suspension criteria and requirements to resume testing
  • Test deliverables

Slide 27:

Content of a Test Plan

Level Test Plan (LTP) Outline per IEEE 829-2008

• Test Management
  • Planned activities and tasks
  • Test progression
  • Environment/infrastructure
  • Responsibilities/authorities
  • Interfaces among stakeholders
  • Resources and training
  • Schedules, estimates, and costs
  • Risk(s) and contingencies
• General
  • Quality assurance procedures
  • Metrics for specific measures
  • Glossary
  • Document change procedures and history

Slide 28:

Content of a Test Plan

Sample Outline of Test Design as Per IEEE 829-2008

(Extended Text Description: Author's relevant description: Sample Outline of Test Design as Per IEEE 829-2008 - A table with four columns of Test design is shown at right side. First row shows Communes: Req. ID; Req.; Test Case ID and Test Case. Second row shows an example, third row show test case. The outline to the left highlights Features to be tested in the sample table and includes:

1. Introduction

1.1. Document identifier

1.2. Scope

1.3. References

2. Details of the Level Test Design

2.1. Features to be tested

2.2. Approach refinement?

2.3. Test identification

2.4. Feature pass/fail criteria 2.5 Test deliverables

3. General

3.1. Glossary

3.2. Document change procedures.)

Slide 29:

Content of a Test Plan

Sample Outline of Test Case as Per IEEE 829-2008

Test Case
ID: TCx.x
Objective:	State which requirement(s) will be verified: testing a dialog correct sequence or correct structure and content of data
Inputs:	Input variable needed for testing
Outcome(s):	Expected results-behavior, errors
Environmental Needs	Test Set Up
Intercase Dependencies	Test cases that must be executed prior to this test case

Slide 30:

Content of a Test Plan

Sample Outline of a Test Procedure from NTCIP 1203 v03

Step	Test Procedure	Results	Additional References
1	CONFIGURE. Determine the maximum period of time that the pixel test should require (based on manufacturer- documentation): RECORD this information as. *Pixel Test Time		:
2	CONFIGURE. Determine the maximum period of time that the message display pixel lest should require (based on manufacturer documentation). RECORD this information as. »Message_Display_Test_ Time
3	SET-UP: Ensure that all pixels are functioning prior to this test.
4	SET the following object(s) to the value(s) shown, »pixelTestActivation.0 = 'test' (3) NOTE-Valid enumerated values are defined in Section 5.11.2.4.3 (Pixel Test Activation Parameter).	Pass / Fail (Section 3.5.3.1.1.2)	Section 4.2.4.2 Step a
5	GET the following object(s). »pixelTestActivation.0	Pass / Fail (RFC 1157)	Section 4.2.4.2 Step b
6	IF the RESPONSE VALUE for pixelTestActivation.0 equals 'test'(3), then GOTO Step 5. Otherwise GOTO Step 7. NOTE-If the RESPONSE VALUE remains at 'test' (3) for more than Pixel Test Time seconds, this test fails.

Slide 31:

Content of a Test Plan

Documentation for Test Reporting

(Extended Text Description: Author's relevant description: Documentation for Test Reporting - The slide shows a layout of three levels of test reporting documentation. At the top is Test Plan Execution box, below is three text boxes appears horizontally-first Level Test Logs report, second box is labeled as Anomaly Report, and last one is level Test Interim test status report. Finally Test Logs and Test Anomaly reports are made part of the final Test report shown at the bottom. This structure is provided by the IEEE 829 standard.)

Slide 32:

Slide 33:

Question

Which is NOT included in a structure of a test plan?

Answer Choices

1. Test logs
2. Test design
3. Test case with inputs/outputs
4. Test procedures with steps

Slide 34:

Review of Answers

a) Test logs
Correct! Test logs are not part of the structure of a test plan. Test logs are developed during and after test execution as part of test reports. This is per the IEEE 829-2008 standard.

b) Test design
Incorrect. The statement is true. Test design provides details on what to test.

c) Test case with inputs/outputs
Incorrect. The statement is true. Test cases detail inputs/outputs.

d) Test procedures with steps
Incorrect. The statement is true. One or more steps are outlined to actually conduct the test.

Slide 35:

Learning Objectives

• Describe within the context of the system lifecycle the role of a test plan and testing to be undertaken
• Recognize the purpose, structure, and content of well-written test documentation for an SSM based on IEEE 829-2008 formats
• Explain how to develop the complete test documentation package for an SSM specification based on NTCIP 1210 Standard v01

Slide 36:

Learning Objective 3

Explain how to develop the complete documentation package for an SSM specification based on NTCIP 1210

Standard v01

Slide 37:

Key Elements of NTCIP 1210 Standard v01 Tied to a Test Plan

Identify Key Elements Used in Preparation of a Test Plan

(Extended Text Description: Author's relevant description: Key Message: This slide makes connection of NTCIP 1210 v01 key elements content to test documents. Each element has a link to test documents. First, user needs/requirements are identified by a Project PRL. Second, objects/dialogs are identified by a project RTM.

Bullet items below connect to Protocol Requirements List (PRL) Module A304a:

• User Needs
• Requirements

Bullet items below connect to Requirements Traceability Matrix (RTM) Module A304b:

• Objects
• Dialogs

)

Slide 38:

Key Elements of NTCIP 1210 Standard v01 Tied to a Test Plan

Use the Project PRL to Identify Features to Be Tested

(Extended Text Description: Author's relevant description: Use the Project PRL to Identify Features to Be Tested PRL table with seven columns and several rows is presented. Fifth column is conformance column which has M, Mandatory requirements and O-optional requirements circled. This shows that Mandatory must be selected and Optional is up to the project to select. An arrow points from Optional O to the requirement stated below: 2.5.2 Manage SSLs These features are to be tested to verify capability to upload-download-retrieve data. Must be selected YES. The table is shown below:

User Need ID	User Need	FR ID	Functional Requirement	Conformance	Support	Additional Specifications
2.4.3	Connect Communication Networks			M	Yes/No
		3.3.1.6	Explore SSL Data by the TMS	M	Yes/No
2.5.2	Manage SSLs			O	Yes/No
		3.3.1.6 ;	Explore SSL Data by the TMS	M	Yes/No
		3.3.1.7 :	TMS Acceptance of Data from the SSL	M	Yes/No
		3.3.1.8	TMS Delivery of Data to the SSL	M	Yes/No

Module A304a

2.5.2 Manage SSLs These features are to be tested to verify capability to upload-download-retrieve data. Must be selected YES.)

Slide 39:

Key Elements of NTCIP 1210 Standard v01 Tied to a Test Plan

Use the Project RTM to Identify Objects to Be Verified

(Extended Text Description: Author's relevant description: Use the Project RTM to Identify Objects to be Verified RTM table with six columns are shown in first row. Three rows below that are requirements. An arrow from object list points to 5.2.1 Maximum Number of SSLs; another arrow points from dialog 4.2.2.3 points to the text below. This slide reminds you with an Icon called checklist shown at corner. The table is shown below:

Functional Requirement Reference	Functional Requirement	Dialog Reference	Object Reference	Object	Comments (Informative)
3.4.2.1	Synchronize SSM Clock with TMS	4.1.3	1201.2.4.1	globalTime
3.4.2.2.1	Determine SSLs Currently Connected	4.2.2.3	5.2.1	maxIntersections
			5.2.2.1.3	intersectionSection
3.4.2.2.2	Determine Pattern Selection Capabilities	4.2.6.3	5.1.1	maxSections
			5.23.1	algorithmSupport

Dialog

Module A304b

4.2.2.3 SSM Intersection / Section Assignment Dialog

The standardized dialog for a TMS to determine the SSLs assigned to a Section within an SSM shall be as follows:

5.2.1 Maximum Number of Intersections SSLs

maxlntersections OBJECT-TYPE

SYNTAX INTEGER (8,-255)

A Test Case Will Be Created Using RTM to Verify Range Values.)

Slide 40:

Developing Test Design, Test Cases, and Test Procedures

Develop an SSM Test Plan

• Describes the Scope, Approach, Resources, and Schedule for the testing
• Some of the testing aspects covered: ° Item(s) to be tested
  • Features to be tested
  • Features not to be tested
  • Testing tasks to be performed
  • Personnel responsible for each task
  • Risks associated with the plan

Slide 41:

Developing Test Design, Test Cases, and Test Procedures

Develop an SSM Test Design Using PRL

• Specifies the detailed approach (design) for exercising a collection of tests
• Identifies the features to be tested by the test design
• Identifies the requirements to be tested by the test design
• Identifies the tests (test cases) associated with the design

Slide 42:

Developing Test Design, Test Cases, and Test Procedures

Develop SSM Test Cases Using Project PRL/RTM

SSM Test Cases

• Defines a test case identified by a test design specification
• Input and output specifications

Slide 43:

Developing Test Design, Test Cases, and Test Procedures

Developing SSM Test Procedures

SSM Test Procedures specify the steps for executing one or more test cases

Slide 44:

Developing Test Design, Test Cases, and Test Procedures

Test Case for Intersection Unit Control Status

(See Module T203 Parts 1 and 2 for Formats)

(Extended Text Description: Author's relevant description: Example of a Test Case for Intersection Unit Control Status - First column of the test case shows TC ID, Objectives, Inputs, outcomes, environmental needs, special requirements, and inter-case dependencies. Second column is populated with relevant details. Table is shown below:

ID: TC001	Title: Request Status Condition within the Device Dialog Verification (Positive Boundary Test Case)
Objective:	To verify system interface implements (positive test case) requirements for a sequence of OBJECT requests for: The test case verifies that the SSM returns an appropriate value given valid data content for the OBJECTs requested at valid value ranges. An output specification is provided, showing valid value constraints per the NTCIP 1210 v01 object definitions.
Inputs: INTEGER { other (1), systemControl (2), systemStandby (3), backupMode(4), manual (5), timebase (6), interconnect (7), interconnectBackup (8)	Step through each object and set a value at the valid value range for the object. For example: Set object 5.8.1.1.5 intersectionUnitControlStatus to '1' (which is just at the valid value range of 1 to 8 inclusive) Set object 5.8.1.1.5 intersectionUnitControlStatus to '8' (which is just at the valid value range of 1 to 8 inclusive)
Outcome(s):	The SSM responds with valid status objects. See Test Case Output Specification TCOS001 - Status Condition within the Device (Boundary Positive Test Case)
Environmental Needs:	No additional needs outside of those specified in the test procedure
Special Procedural Requirements:	None
Intercase Dependencies:	None

Note: overlapped on this table is another small table:

5.8.1.1.5	intersectionUnitControlStatus
5.8.1.1.6	intersectionCurrentEventLogSize

)

Slide 45:

Develop Requirements to Test Case Traceability Matrix (RTCTM) for an SSM

Developing an SSM RTCTM

• An RTCTM is a table that provides traceability from requirements to test cases to test procedures
• Each SSM Test Design (Test Plan) has an RTCTM

(Extended Text Description: Author's relevant description: Developing SSM RTCTM Six column RTCTM is shown with rows. The table is shown below:

Req. ID	Req.	Test Case ID	Test Case	Test Proc ID	Test Procedure
3.4.2.2.1	Explore SSL Data by the TMS
		TC3.4.3.1.6-1	Verify maximum intersections
				TP3.4.3 .1.6-1	Verify object range 8-40

)

Slide 46:

Develop Requirements to Test Case Traceability Matrix (RTCTM) for an SSM

RTCTM Lists Test Procedures for Each Test Case

• RTCTM has one or more Test Cases to verify conformance to NTCIP 1210 v01
• RTCTM lists one or more Test Procedures to verify object range

(Extended Text Description: Author's relevant description: RTCTM Lists Test Procedures for Each Test Case – Fifth and sixth columns contains Test Procedures which is highlighted with a box. The table is the same table as slide 45.)

Slide 47:

Develop Requirements to Test Case Traceability Matrix (RTCTM) for an SSM

Test Case/Test Procedures

(See Module T204 Parts 1 and 2)

Test Case: TC1.1

Title:	Test the Boundaries
Description	This test case verifies the maximum number of SSMs that can be SET by the central station. The test is conducted just below, just above, and exactly at the boundary.
Variables	Max SSMs	From project requirements
	Max SSMs -1	From the test plan
	Max SSMs +1	From the test plan
Pass/Fail Criteria	1. The DUT shall accept data at Max SSMs 2. The DUT shall accept data at Max SSMs -1 3. The DUT shall return an error at Max SSMs +1

Steps are formal executions and results oriented (must have an outcome)

Step	Test Procedure	Expected Results
1	Configure: SET the Max SSMs = 8, record the DUT response	Responds with Max SSMs = 8
2	SET the number of SSMs = 1, record the DUT response	Response =1
3	SET the number of SSMs = 2, record the DUT response	Response = 2
4	SET the number of SSMs = 10, record the DUT response	Error, exceeds Max SSMs = 8

Slide 48:

Introduction to the Test Procedure Generator (TPG) and How to Use It for SSM Testing

Test Procedure Generator (TPG)

• TPG is a software that guides the development of the test procedures
• Used for Center to Field (C2F) devices
• Relatively new product
• v2.1 downloadable at:
  https://www.standards.its.dot.gov/DeploymentResources/TPGdownload

Slide 49:

Introduction to the Test Procedure Generator (TPG) and How to Use It for SSM Testing

How to Use the Test Procedure Generator (TPG)

• Install the TPG Software
• Import the Standard, and the TPG will process the requirements, objects, dialogs, and RTM
• Create "Set of Test Procedures" (Note: This feature will allow a user to begin to develop test procedures)
• Develop Test Procedures (covered in detail on the next slide)

Slide 50:

Introduction to the Test Procedure Generator (TPG) and How to Use It for SSM Testing

Test Procedure with the TPG by User

• Defines the title, description, and pass/fail criteria in the header information
• Selects the requirements to be tested (TPG has imported the list from the standard)
• Creates the variables to be used in the test procedure (the TPG uses the objects imported from the standard)
• Develops the detailed steps using the TPG tools

Slide 51:

Introduction to the Test Procedure Generator (TPG) and How to Use It for SSM testing

TPG Benefits

• Test procedures come from the agency specification, NOT from vendors:
  • Reduces developmental risks, effort, and the cost
  • Ensures traceability, and conformance to the Center to Field (C2F) Standards such as DMS, ESS, and SSM
  • Helps determine compliance to extended standard
  • Promotes interoperable C2F systems
• Creates in-house expertise

Slide 52:

Slide 53:

Question

What is the primary purpose of RTCTM? Answer Choices

1. Sets the testing workflow sequences
2. Correlates User Needs to Requirements
3. Contains only test cases
4. Traces Requirement to Test Case to Test Procedure

Slide 54:

Review of Answers

a) Sets the testing workflow sequences
Incorrect. Testing workflow is part of the Level Test Plans.

b) Correlates User Needs to Requirements
Incorrect. User Needs to Requirements are part of the Protocol Requirements List (PRL).

c) Contains only test cases
Incorrect. It contains test cases and test procedures for each test case.

d) Traces Requirement to Test Case to Test Procedures
Correct! RTCTM depicts the Test Cases that will be used to verify each Requirement with test procedures.

Slide 55:

Learning Objectives

• Describe within the context of the system lifecycle the role of a test plan and testing to be undertaken
• Recognize the purpose, structure, and content of well-written test documentation for an SSM based on IEEE 829-2008 formats
• Explain how to develop the complete test documentation package for an SSM specification based on NTCIP 1210 Standard v01
• Describe the testing of an SSM using sample test document

Slide 56:

Learning Objective 4

• Describe the Testing of an SSM Using Sample Test Document

Slide 57:

Walk Through a Sample SSM Test Plan

Where Is the SSM Test Plan Located?

General Procurement Contract Documents

Communications Interface Specifications

I. General

II. SSM User Needs

III. SSM Functional Req.

IV. SSM Project PRL, RTM

V. Testing Documentation

Slide 58:

Walk Through a Sample SSM Test Plan

Description of an SSM Testing Setup

(Extended Text Description: Author's relevant description: Description of an SSM Testing Set Up - A Test Plan text box at left is point to a middle set up of SSM under test with arrow. From the test set up an arrow comes out and points to Test results. The test set up in the middle has a PC, laptop and Device under test-SSM. Key Message: This slide explains what an SSM test setup is and what we get out of it. The test plan drives the testing process, and testing results are reported, including anomaly reports.)

Slide 59:

Walk Through a Sample SSM Test Plan

How Is the SSM Test Plan Developed?

Test documentation is developed for a given project using IEEE Std 829-2008 formats

(Extended Text Description: Author's relevant description: How Is the SSM Test Plan Developed? - On the right a Test Plan based on IEEE 829 format is shown. The outline is same as one describe in Slide 25. Text on the left points to the test plan on the right: Test documentation is developed for a given project using IEEE Std 829-2008 formats. Test Design and a Test Plan can be in one document for a single test design. Test Cases and Test Procedures can be combined in one document.)

Slide 60:

Walk Through a Sample SSM Test Plan

(Extended Text Description: Author's relevant description: Test Plan Outline - A Test Plan text box is shown on right side with text on the left pointing to the text Forms basis for what to test:

Test Plan Outline

Key Parts

1. Introduction

2. Details of Unit Testing

2.1 Test items and their identifiers

2.2 RTCTM (Test Design/Test Procedures)

2.3 List of SSM Features to be tested (PRL)

2.4 Objects to be tested (RTM)

2.5 Approach

2.6 Item Pass/Fail criteria

2.7 Suspension Criteria and Resumption Requirements

)

Slide 61:

Walk Through a Sample SSM Test Plan

Test Plan Outline

Key Parts (cont.)

2.8 Test Deliverables (Before Testing)

SSM Communication Test Plan
SSM Communication Test Designs
SSM Communication Test Cases
SSM Communication Test Procedures

Reporting Results (During/After Testing)

SSM Communication Test Logs
SSM Communication Test Incident Reports
SSM Communication Interim Test Status Reports
SSM Communication Test Reports

Slide 62:

Slide 63:

Walk Through a Sample SSM Test Plan

The City of Midsize: SSM Communications Interface Specification

• Central TMS requires NTCIP 1210 v01 communications interface with response time of 600 msecs.
• One SSM monitors/controls maximum of 10 SSLs located.
• Traffic responsive strategy covers 30 SSLs spread over three sections.
• Existing communication interconnect is declared adequate for the controllers.
• Project PRL and RTM are also included in the specification.

Slide 64:

Walk Through a Sample SSM Test Plan

What Are the Project Parameters?

(Extended Text Description: Author's relevant description: What Are the Project Parameters? - A Layout of SSM and SSLs is shown to the right of list of parameters on left. In the example network in this diagram, we can see that 3 SSMs will be needed to control 10 intersections, for 3 sections.)

Slide 65:

Walk Through a Sample SSM Test Plan

PRL Example: What Needs to Be Tested/Not to Be Tested

(Extended Text Description: Author's relevant description: PRL Example: What Needs to Be Tested/Not to be Tested - PRL example for 3.4.2 is shown with a red box over requirements with what will be tested with YES marked with circle and what will not be tested. M-is always tested and O-is optional which is selected by project. The table is shown below:

User Need ID	User Need	FR ID	Functional Requirement	Conformance	Support	Additional Specifications
2.5.1.1	Configure Cycle Timers and Unit Backup Time			M	Yes
		3.4.2 Manage the SSM Configuration
		3.4.2.2.1	Determine SSLs Currently Connected:	M	Yes
		3.4.2.2.2	Determine Pattern Selection Capabilities...........:.....	M	Yes	.... will be tested
		3.4.2.2.3	Determine :SSM Section Characteristics	M	Yes
		3.4.2.2.4.1	Configure Cycle Timer Reference	O	Yes / No
		3.4.2.2.4.2	Determine Cycle Timer Capability	O	Yes / No	.... will NOT be tested
		3.4.2.2.5	Determine SSM Software Version	M	Yes / No

)

Slide 66:

Walk Through a Sample SSM Test Plan

Find Object Ranges from Project RTM to Prepare Test Cases

(Extended Text Description: Author's relevant description: Find Object Ranges from Project RTM to Prepare Test Cases - An arrow is points to 8-255 crossed with red line in object range. The case study has 10 SSLs to be tested. The table is shown below:

Functional Requirement Reference	Functional Requirement	Dialog Reference	Object Reference	Object	Comments (Informative)
3.4.2.1	Synchronize SSM Clock with TMS	4.1.3	1201.2.4.1	globalTime
3.4.2.2.1	Determine SSLs Currently Connected	4.2.2.3	5.2.1	maxlntersections
			5.2.2.1.3	intersectionSection
3.4.2.2.2	Determine Pattern Selection Capabilities	4.2.6.3	5.1.1	maxSections
			5.23.1	algorithmSupport

5.2.1 Maximum Number of Intersections

maxIntersections OBJECT-TYPE

SYNTAX INTEGER (8..255)

Recall, case study has 10 SSLs requirement here.)

Slide 67:

Walk Through a Sample SSM Test Plan

Prepare RTCTM for Testing Documentation

(Extended Text Description: Author's relevant description: Prepare RTCTM for Testing Documentation - Example show 10 in Test Procedure column. The table is shown below:

Req. ID	Req.	Test Case ID	Test Case	Test Proc ID	Test Procedure
3.4.2.2.1	Explore SSL Data by the TMS
		TC3.4.3 .1.6-1	Verify maximum intersections
				TP3.4.3 .1.6-1	Verify object range 10

)

Test Procedure will be carried out to check boundary condition at 10, just below at 8, and just above at 12

Slide 68:

Results-Error Conditions

Testing for Boundary Conditions

• All boundary conditions are tested:
  • Just below each limit
  • Just above each limit
  • Exactly on each limit
• Boundary is valid, SSM should:
  • Process successfully
  • Respond accordingly
• If error conditions occur, SSM should:
  • Respond with error message
  • Remain in normal operation
  • No communications loss

Slide 69:

Results-Error Conditions

How Are we Checking for Error Conditions?

• Positive testing for:
  • Validating input values, dialogs, and sequences per test procedure
  • Expected outputs from SSM - Device Under Test (DUT)
• Negative testing for:
  • Asserting invalid input values, dialogs, or sequences per the test procedure
  • Errors are examined for next action on test continuity

Slide 70:

Results-Error Conditions

Testing for Value Outside Valid Boundary Range

(Extended Text Description: Author's relevant description: Example: Testing for Value Outside Valid Boundary Range Test Case with columns is shown.

ID: TC001	Title: Request Status Condition within the Device Dialog Verification (Positive Boundary Test Case)
Objective:	To verify system interface implements (positive test case) requirements for a sequence of OBJECT requests for: The test case verifies that the SSM returns an appropriate value given valid data content for the OBJECTs requested at valid value ranges. An output specification is provided, showing valid value constraints per the NTCIP 1210 v01 object definitions.
Inputs: INTEGER { other (1), systemControl (2), systemStandby (3), backupMode(4), manual (5), timebase (6), interconnect (7), interconnectBackup (8)	Step through each object and set a value at the valid value range for the object. For example: Set object 5.8.1.1.5 intersectionUnitControlStatus to '1' (which is just at the valid value range of 1 to 8 inclusive) Set object 5.8.1.1.5 intersectionUnitControlStatus to '8' (which is just at the valid value range of 1 to 8 inclusive)
Outcome(s):	The SSM responds with valid status objects. See Test Case Output Specification TCOS001 - Status Condition within the Device (Boundary Positive Test Case)
Environmental Needs:	No additional needs outside of those specified in the test procedure
Special Procedural Requirements:	None
Intercase Dependencies:	None

Note: overlapped on this table is another small table:

5.8.1.1.5	intersectionUnitControlStatus
5.8.1.1.6	intersectionCurrentEventLogSize

)

Slide 71:

Results-Error Conditions

(Extended Text Description: Author's relevant description: PRL Example: What Needs to be Tested: Mandatory Requirements PRL shows a red box over conformance and support columns. All of them are to be tested. Table is shown below:

PRL Example: What Needs to Be Tested: Mandatory Requirements

User Need ID	User Need	FR ID	Functional Requirement	Conformance	Support	Additional Specifications
2.4.1	Provide Live Data			M	Yes
		3.3.1.1	Accept Data from the TMS	M	Yes	All are to be test
		3.3.1.2	Deliver Data to the TMS	M	Yes
		3.3.1.3	Explore SSM Data by the TMS	M	Yes
		3.3.3.1	Determine Access Settings	M	Yes
		3.3.3.2	Configure Access	M	Yes

)

Slide 72:

Testing Tools

Communications Testing Tools Available

• Many generic Simple Network Management Protocol (SNMP) test tools available for Ethernet communications
• Data Analyzers
• NTCIP Testing Tools
  • Test both Ethernet and serial communications
  • Test all objects within the MIB with Set/Get operations
  • Verify that read-only objects are not settable
  • Logs and reports (various levels)

Slide 73:

Testing Tools

Where to Find Additional Test Procedure Information

Additional Information on Test Procedures:

• NTCIP 1203 v03 DMS Standard, Annex C
• NTCIP 1204 v03 ESS Standard, Annex C
• Module T313: Applying Your Test Plan to NTCIP 1204 ESS
• Test Procedure Generator (TPG v2.1)

Slide 74:

Slide 75:

Question

Which is NOT a valid statement related to an SSM testing documentation?

Answer Choices

1. Test plan contains an overall testing approach
2. Test design contains project RTCTM
3. Test procedures are provided by the manufacturer
4. Test procedure includes error detection

Slide 76:

Review of Answers

a) Test plan contains an overall testing approach
Incorrect. The statement is true. A plan has an overall approach and scope.

b) Test design contains project RTCTM
Incorrect. RTCTM correlates requirements, test cases, and set procedures to verify a requirement.

c) Test procedures are provided by the manufacturer
Correct! The statement is NOT true. ONLY agency specification specifies test procedures.

d) Test procedure includes error detection
Incorrect. The statement is true. The test includes both positive and negative testing for expected and unexpected results, respectively.

Slide 77:

Module Summary

• Describe within the context of the system lifecycle the role of a test plan and testing to be undertaken
• Recognize the purpose, structure, and content of well-written test documentation for an SSM based on IEEE 829-2008 formats
• Explain how to develop the complete test documentation package for an SSM specification based on NTCIP 1210 Standard v01
• Describe the testing of an SSM using sample test document

Slide 78:

We Have Now Completed the SSM Curriculum

Module A304a:
Understanding User Needs for Field Management Stations - Part 1 Object Definitions for Signal System Masters (SSM) Based on NTCIP 1210 Standard

Module A304b:
Specifying Requirements for Field Management Stations - Part 1 Object Definitions for Signal System Masters (SSM) Based on NTCIP 1210 Standard

Module T304:
Applying Your Test Plan to Field Management Stations - Part 1 Signal System Masters (SSM) Based on NTCIP 1210 Standard v01

Slide 79:

Thank you for completing this module.

Feedback

Please use the Feedback link below to provide us with your thoughts and comments about the value of the training.

Thank you!