Module 35 - T315

T315: Applying Your Test Plan to the NTCIP 1202 ASC Standard

HTML of the PowerPoint Presentation

(Note: This document has been converted from a PowerPoint presentation to 508-compliant HTML. The formatting has been adjusted for 508 compliance, but all the original text content is included, plus additional text descriptions for the images, photos and/or diagrams have been provided below.)

 

Slide 1:

Welcome - Graphic image of introductory slide. Please see the Extended Text Description below.

(Extended Text Description: Slide 1: Welcome - Graphic image of introductory slide. A large dark blue rectangle with a wide, light grid pattern at the top half and bands of dark and lighter blue bands below. There is a white square ITS logo box with words "Standards ITS Training" in green and blue on the middle left side. The word "Welcome" in white is to the right of the logo. Under the logo box are the words "RITA Intelligent Transportation Systems Joint Program Office.")

 

Slide 2:

Welcome

Head shot photo of Ken Leonard, Director - ITS Joint Program Office

Ken Leonard, Director

ITS Joint Program Office

Ken.Leonard@dot.gov

Screen capture snapshot of RITA website - for illustration only - see the extended text description below.

(Extended Text Description: Intro Slide: Screen capture snapshot of RITA website - for illustration only. Below this image is a link to the current website: http://www.pcb.its.dot.gov - this screen capture snapshot shows an example from the RITA website from April 2013. At the top of the page it shows the RITA logo with the text U.S. Department of Transportation Research and Innovative Technology Administration - Intelligent Transportation Systems Joint Program Office - ITS Professional Capacity Building Program/Advanced ITS Education. Below the main site banner, it shows the main navigation menu with the following items: About, ITS Training, Knowledge Exchange, Technology Transfer, ITS in Academics, and Media Library. Below the main navigation menu, the page shows various content of the website, including a graphic image of professionals seated in a room during a training program. A text overlay has the text Welcome to ITS Professional Capacity Building. Additional content on the page includes a box entitled What's New and sections entitled Available E-Training (free), Free ITS Training and T3 Webinars. Again, this image serves for illustration only. The current website link is: http://www.pcb.its.dot.gov)

www.pcb.its.dot.gov

(Note: There is additional text attached to this slide that includes the following introductory information from Ken Leonard):

"ITS Standards can make your life easier. Your procurements will go more smoothly and you'll encourage competition, but only if you know how to write them into your specifications and test them. This module is one in a series that covers practical applications for acquiring and testing standards-based ITS systems.

I am Ken Leonard, director of the ITS Joint Program Office for USDOT and I want to welcome you to our newly redesigned ITS standards training program of which this module is a part. We are pleased to be working with our partner, the Institute of Transportation Engineers, to deliver this new approach to training that combines web based modules with instructor interaction to bring the latest in ITS learning to busy professionals like yourself.

This combined approach allows interested professionals to schedule training at your convenience, without the need to travel. After you complete this training, we hope that you will tell colleagues and customers about the latest ITS standards and encourage them to take advantage of the archived version of the webinars.

ITS Standards training is one of the first offerings of our updated Professional Capacity Training Program. Through the PCB program we prepare professionals to adopt proven and emerging ITS technologies that will make surface transportation safer, smarter and greener which improves livability for us all. You can find information on additional modules and training programs on our web site www.pcb.its.dot.gov

Please help us make even more improvements to our training modules through the evaluation process. We look forward to hearing your comments. Thank you again for participating and we hope you find this module helpful."

 

Slide 3:

Activity. A placeholder graphic with an image of hand over a computer keyboard to show that an activity is taking place.

 

Slide 4:

T315:

Applying Your Test Plan to the NTCIP 1202 ASC Standard

 

Slide 5:

Instructor

Head shot photo of Kenneth L. Vaughn, P.E. - President - Trevilon LLC - Magnolia, TX, USA

Kenneth L. Vaughn, P.E.
President
Trevilon LLC
Magnolia, TX, USA

 

Slide 6:

Target Audience

 

Slide 7:

Recommended Prerequisite(s)

 

Slide 8:

Curriculum Path

A graphical illustration indicating the sequence of training modules that lead up to and follow this course. Please see the Extended Text Description below.

(Extended Text Description: A graphical illustration indicating the sequence of training modules that lead up to and follow this course. Each module is represented by a box with the name of the module in it and an arrow showing the logical flow of the modules and the current module highlighted. This slide focuses on the modules that lead up to the current course. The first box is labeled "T101 Introduction to ITS Standards Testing." An arrow from this box connects it to a box labeled "T201 How to Write a Test Plan." An arrow from this box connects it to a box labeled "T202 Overview of Test Design Specifications, Test Cases, and Test Procedures." An arrow from this box connects it to a box located at the start of the next line and labeled "C101 Introduction to the Comm Protocols and Their Uses in ITS Applications." An arrow from this box connects it to a box labeled "A315a Understanding User Needs for ASC Based on NTCIP 1202 Standard." An arrow from this box connects it to a box labeled "A315b Understanding Requirements for ASC Based on NTCIP 1202 Standard." Finally, an arrow from this box connects it to a highlighted box designating this module, which is located at the start of the next line and labeled "T315 Applying Your Test Plan (ASC) to NTCIP 1202 ASC Standard.")

 

Slide 9:

Learning Objectives

  1. Recognize the importance of testing ASCs
  2. Apply the rules for developing a sample ASC test plan
  3. List the rules for developing test case specifications and procedures for NTCIP 1202
  4. Develop sample test case specifications and procedures for NTCIP 1202
  5. Understand testing results for NTCIP 1202

 

Slide 10:

Learning Objective #1 — Recognize the Importance of Testing ASCs

 

Slide 11:

Learning Objective #1

Recognize the Importance of Testing

Safety Implications

 

Slide 12:

Learning Objective #1

Recognize the Importance of Testing

Complexity Issues

 

Slide 13:

Learning Objective #1

Recognize the Importance of Testing

Functional vs. Communication Testing

What is NTCIP testing?

Both are defined by NTCIP, but test plan may separate for convenience

 

Slide 14:

Learning Objective #1

Recognize the Importance of Testing

Repeatability

 

Slide 15:

Learning Objective #1

Recognize the Importance of Testing

Budgeting

 

Slide 16:

Activity. A placeholder graphic with an image of hand over a computer keyboard to show that an activity is taking place.

 

Slide 17:

Learning Objective #1

Why is it important to test ASCs?

  1. Implementation errors might result in conflicting green displays
  2. Implementation errors might decrease traveler safety
  3. Testing is not important for ASCs
  4. It gives peace of mind for a trivial cost

 

Slide 18:

Learning Objective #1

Review of Answers

A small graphical red and yellow X representing incorrect.a) Implementation errors might result in conflicting green displays
Incorrect; a properly configured conflict monitor will ensure that conflicting displays are never shown.

A small graphical green and yellow check mark representing correct.b) Implementation errors might decrease traveler safety
Correct! An error can result in an undesired configuration and decrease safety.

A small graphical red and yellow X representing incorrect.c) Testing is not important for ASCs
Incorrect; testing is especially important for ASCs due to their ability to directly control traffic operations.

A small graphical red and yellow X representing incorrect.d) It gives peace of mind for a trivial cost
Incorrect; proper testing requires significant effort.

 

Slide 19:

Summary of Learning Objective #1

Recognize the Importance of Testing ASCs

 

Slide 20:

Learning Objective #2 — Develop a Sample Test Plan

 

Slide 21:

Learning Objective #2

Develop a Sample Test Plan

Test Documentation

 

Slide 22:

Learning Objective #2

Develop a Sample Test Plan

Identify Requirements to Test

 

Slide 23:

Learning Objective #2

Develop a Sample Test Plan

Identify Test Phase

 

Slide 24:

Learning Objective #2

Develop a Sample Test Plan

Identify Test Methodology

 

Slide 25:

Learning Objective #2

Develop a Sample Test Plan

Requirements to Test Case Traceability Matrix

Req't ID Requirement TC ID Test Case
2.2.1.1 Configure a Timing Pattern 2.3.1.1 Configure a Valid Timing Pattern
2.3.1.2 Incorrectly Configure a Timing Pattern
2.2.4.1 Support at least 32 Timing Patterns 2.3.1.1 Configure a Valid Timing Pattern
2.3.1.3 Configure Timing Pattern 32
2.2.1.2 Configure Timing Pattern Selection Logic See Manufacturer Factory Acceptance Test Plan
2.2.2.1 Activate Timing Pattern Remotely 2.3.1.5 Activate Timing Pattern Remotely
2.2.2.2 Activate a Timing Pattern per a Schedule 2.3.1.6 Activate a Timing Pattern per a Schedule

* Req't and TC ID reference clauses in the Participant Student Supplement

 

Slide 26:

Learning Objective #2

Develop a Sample Test Plan

Identify Test Environment

A sample test environment as incorporated from NTCIP 8007, Page 13. Please see the Extended Text Description below.

(Extended Text Description: A sample test environment as incorporated from NTCIP 8007, Page 13. A "device under test (DUT)" is connected to a communications cloud which is connected to a "test application" running on a PC. Also connected to the communications cloud is a data analyzer, also running on a PC.)

Source: NTCIP 8007, page 13

 

Slide 27:

Learning Objective #2

Develop a Sample Test Plan

Plan Logistics of Testing

 

Slide 28:

Learning Objective #2

Develop a Sample Test Plan

Estimate Level of Effort

Estimate effort, schedule, and budget for:

 

Slide 29:

Learning Objective #2

Develop a Sample Test Plan

Evaluate Risks

 

Slide 30:

Learning Objective #2

Develop a Sample Test Plan

Understanding the Impact of a Failure

Req' t ID Requirement TC ID Test Case
2.2.1.1 Configure a Timing Pattern 2.3.1.1 Configure a Valid Timing Pattern
2.3.1.2 Incorrectly Configure a Timing Pattern
2.2.4.1 Support at least 32 Timing Patterns 2.3.1.1 Configure a Valid Timing Pattern
2.3.1.3 Configure Timing Pattern 32
2.2.1.2 Configure Timing Pattern Selection Logic See Manufacturer Factory Acceptance Test Plan
2.2.2.1 Activate Timing Pattern Remotely 2.3.1.5 Activate Timing Pattern Remotely
2.2.2.2 Activate a Timing Pattern per a Schedule 2.3.1.6 Activate a Timing Pattern per a Schedule
H.2.2.1 Set Time 2.3.3.1 Set Time

* Req't and TC ID reference clauses in the Participant Student Supplement

 

Slide 31:

Learning Objective #2

Develop a Sample Test Plan

Understanding the Impact of a Failure

UN ID User Need RID Requirement
2.1.3.1 Control Selection of Timing Pattern
  2.2.1.1 Configure a Timing Pattern
  2.2.4.1 Support at Least 32 Timing Patterns
  2.2.1.2 Configure Timing Pattern Selection Logic
  2.2.2.1 Activate Timing Pattern Remotely
  2.2.2.2 Activate a Timing Pattern per a Schedule
  2.2.2.3 Override Timing Pattern
  2.2.3.6 Retrieve a Schedule
  2.2.2.8 Define a Schedule
  NTCIP 1203v03 H.2.2.1 Set Time
  NTCIP 1203v03 H.2.2.2 Set Time Zone
  NTCIP 1203v03 H.2.2.3 Set Daylight Savings Mode
  NTCIP 1203v03 H.2.2.4 Verify Current Time
  ... ...

* UN and R ID reference clauses in the Participant Student Supplement

 

Slide 32:

Learning Objective #2

Develop a Sample Test Plan

Plan Project Close-out

 

Slide 33:

Activity. A placeholder graphic with an image of hand over a computer keyboard to show that an activity is taking place.

 

Slide 34:

Learning Objective #2

Which of the following statements is not true?

  1. Every requirement should be tested
  2. Some testing may be performed by the manufacturer
  3. You should only need to perform your test plan once
  4. Traceability tables can help you to assess the impact of a test failure

 

Slide 35:

Learning Objective #2

Review of Answers

A small graphical red and yellow X representing incorrect.a) Every requirement should be tested
True; every requirement should be tested.

A small graphical red and yellow X representing incorrect.b) Some testing may be performed by manufacturer
True; testing may be performed by the agency, the manufacturer, or a third party.

A small graphical green and yellow check mark representing correct.c) You should only need to perform your test plan once
False (correct). Testing will often reveal problems; these should be fixed and the device retested.

A small graphical red and yellow X representing incorrect.d) Traceability tables can help you to assess the impact of a test failure
True; traceability tables allow you to identify the user needs that will not be completely fulfilled.

 

Slide 36:

Summary of Learning Objective #2

Develop a Sample Test Plan

 

Slide 37:

Learning Objective #3 — List the Rules for Developing Test Case Specifications and Procedures for NTCIP 1202

 

Slide 38:

Learning Objective #3

Test Case Specifications and Procedures

Review Guidance

 

Slide 39:

Learning Objective #3

Test Case Specifications and Procedures

Components

NTCIP 8007 describes how these can be combined for NTCIP testing

 

Slide 40:

Test Case Specifications and Procedures

Sample Basic Dialog

A slide showing a sample basic dialog. Please see the Extended Text Description below.

(Extended Text Description: A slide showing a sample basic dialog. It is a UML sequence diagram with a "TMS" sending a "Set(globalTime.0)" message to an "ASC" followed by the "ASC" sending an unnamed response message to the "TMS".)

 

Slide 41:

Learning Objective #3

Test Case Specifications and Procedures

Specify Each Test Case

Req't ID Requirement TC ID Test Case
H.2.2.1 Set Time 2.3.2.1 Set Time
Test Case 2.3.3.1 Title Set Time
Description This test case verifies that the ASC properly tracks time. It advances the clock by a user-defined amount, waits a few seconds, retrieves the time, and verifies it indicates an appropriate value. This test will advance the ASC clock by Time_Offset seconds.
Variables Time_Offset as defined in the test plan.
Pass/Fail Criteria The DUT shall pass every verification step included within the Test Case to pass the Test Case.

NOTE: If requirement is from an SEP standard, you may be able to reference the standard test case as well

 

Slide 42:

Learning Objective #3

Test Case Specifications and Procedures

Specify Procedures

 

Slide 43:

Learning Objective #3

Test Case Specification and Procedures

Sample Procedure

Step Test Procedure Results
1 CONFIGURE: Determine the number of seconds to advance the clock in the ASC. RECORD this information as: Time_Offset  
2 GET the following object(s): globalTime.0 Pass/Fail
(Section H.2.2.4)
3 RECORD the RESPONSE VALUE for globalTime.0 as Start_Time  
4 SET the following object(s):
globalTime.0 = Start_Time + Time_Offset
Pass/Fail
(Section H.2.2.1)
5 DELAY for 15 seconds  
6 GET the following object(s):
globalTime.0
Pass/Fail
(Section H.2.2.4)
7 VERIFY that the RESPONSE VALUE for globalTime.0 is roughly equal to Start_Time + Time_Offset + 15 Pass/Fail
(Section H.2.2.4)

 

Slide 44:

Activity. A placeholder graphic with an image of hand over a computer keyboard to show that an activity is taking place.

 

Slide 45:

Learning Objective #3

Where can you find definitions for terms that can be used in NTCIP test steps?

  1. IEEE 829
  2. ISO 9001
  3. NTCIP 8007
  4. Student supplement

 

Slide 46:

Learning Objective #3

Review of Answers

A small graphical red and yellow X representing incorrect.a) IEEE 829
Incorrect; IEEE 829 defines sample outlines for test documentation, but does not define steps for NTCIP

A small graphical red and yellow X representing incorrect.b) ISO 9001
Incorrect; ISO 9001 deals with quality management, but does not deal directly with NTCIP testing

A small graphical green and yellow check mark representing correct.c) NTCIP 8007
Correct! NTCIP 8007 defines a number of terms that can be used in test steps for NTCIP testing

A small graphical red and yellow X representing incorrect.d) Student Supplement
Incorrect; the student supplement provides samples of test procedures, but it does not define the test terms.

 

Slide 47:

Summary of Learning Objective #3

List the Rules for Developing Test Case Specifications and Procedures for NTCIP 1202

 

Slide 48:

Learning Objective #4 — Develop Sample Test Case Specifications and Procedures for NTCIP 1202

 

Slide 49:

Develop Sample Test Cases and Procedures Database Transactions

 

Slide 50:

Learning Objective #4

Develop Sample Test Cases and Procedures Database Transactions

 

Slide 51:

Learning Objective #4

Database Transactions

Sample User Need

Agencies need to be able to fully configure an ASC remotely. Fully configuring an ASC requires setting a large number of inter-related parameters that may not fit into a single request and may require considerable time to validate; however, the entire configuration must be implemented simultaneously due to their interrelationships.

 

Slide 52:

Learning Objective #4

Database Transactions

Sample Requirement

Until all inter-related database parameters have been downloaded, the ASC shall not implement new values for any database parameter but will process operations on non-database parameters normally.

 

Slide 53:

Learning Objective #4

Database Transactions

Sample Dialog Part 1

A slide showing part 1 of the sample dialog for database transactions. Please see the Extended Text Description below.

(Extended Text Description: A slide showing part 1 of the sample dialog for database transactions. It is depicted by a UML sequence diagram between a "TMS" and an "ASC". The dialog starts with the "TMS" sending a "Get(dbCreateTrtansaction.0)" message to the "ASC". This is followed by the "ASC" sending an unlabeled response message to the "TMS". If the response is normal, the sequence continues and the "TMS" sends a "Set(dbCreateTransaction = transaction)" message to the "ASC", which responds with an unnamed response. The "TMS" then enters a loop, designated with a box labeled "repeat as needed in any order" which includes: 1. The "TMS" sends a "Set(any combination of writable objects to a valid values") and the "ASC" responds with a response message. 2. The "TMS" sends a "Get(any object)" message and the "ASC" responds with a response message.)

 

Slide 54:

Learning Objective #4

Database Transactions

Sample Dialog Part 2

A slide showing part 2 of the sample dialog for database transactions. Please see the Extended Text Description below.

(Extended Text Description: A slide showing part 2 of the sample dialog for database transactions. It is depicted by a UML sequence diagram between a "TMS" and an "ASC". The dialog continues with the "TMS" sending a "Set(dbCreateTrtansaction.0 = verify)" message and the "ASC" responds with a response message with a note that the ASC will "respond and then perform consistency checks". The "TMS" then enters a loop, designated with a box labeled "repeat until dbCreateTransaction.0 = done"; the box contains: 1. The "TMS" sending a "Get(dbCreateTransaction.0)" message and the "ASC" responding with a response message. The dialog then continues with the "TMS" sending a "Get(dbVerifyStatus.0, dbVerifyError.0)" message and the "ASC" responding with a response message. The "TMS" then sends a "Set(dbCreateTransaction.0 = normal)" message and the "ASC" responds with a response message.)

 

Slide 55:

Learning Objective #4

Database Transactions

Sample Test Cases

 

Slide 56:

Learning Objective #4

Database Transactions

Sample Test Case

Test Case 6.1 Title Download a valid database configuration
Description This test case verifies that the ASC properly buffers database parameters until all inter-related parameters are downloaded. It initializes, starts the database transaction, downloads a number of parameters, retrieves the parameters, verifies that they are unchanged, completes the transaction, retrieves the parameters, and verifies that they have been updated.
This test modifies the database configuration for the ASC.
Variables Phase_Startup_1, ...
Phase_Options_1, ...
Phase_Ring_1, ...
...
Pass/Fail Criteria The DUT shall pass every verification step included within the Test Case to pass the Test Case.

 

Slide 57:

Learning Objective #4

Database Transactions

Sample Procedure - Initialize and Start

Step Test Procedure Results
1 CONFIGURE: Determine the desired state of phase 1 at startup. RECORD this information as: Phase_Startup_1  
... ...  
100 GET the following object(s):
phaseStartup.1
phaseStartup.2
...
Pass/Fail
(NTCIP 1202 - 2.2.2.20)
101 RECORD phaseStartup.1 as Orig_Phase_Startup_1  
... ... ...
200 GET the following object(s):
dbCreateTransaction.0
Pass/Fail
(NTCIP 1201 -2.3.1)
201 SET-UP: VERIFY that the value of dbCreateTransaction.0 is equal to 'normal' (1).  
202 SET the following object(s) to the values shown:
dbCreateTransaction.0 = 'transaction' (2)
Pass/Fail
(NTCIP 1201 -2.3.1)

 

Slide 58:

Learning Objective #4

Database Transactions

Sample Procedure - Download

Step Test Procedure Results
203 SET the following object(s) to the values shown:
phaseStartup.1 = Phase_Startup_1
phaseStartup.2 = Phase_Startup_2
...
Pass/Fail
(NTCIP 1201 -2.3.1;
NTCIP 1202 -2.2.2.20)
... ... ...
300 GET the following object(s) :
phaseStartup.1
phaseStartup.2
...
Pass/Fail
(NTCIP 1202 - 2.2.2.20)
301 VERIFY that phaseStartup.1 equals Orig_Phase_Startup_1 Pass/Fail
(NTCIP 1201 -2.3.1)
... ... ...
400 SET the following object(s) to the values shown:
dbCreateTransaction.0 = 'verify' (3)
Pass/Fail
(NTCIP 1201 -2.3.1)

 

Slide 59:

Learning Objective #4

Database Transactions

Sample Procedure - Verify

Step Test Procedure Results
401 GET the following object(s) :
dbCreateTransaction.0
Pass/Fail
(NTCIP 1201 -2.3.1)
402 IF dbCreateTransaction.0 equals 'done', GOTO Step 403; otherwise GOTO Step 401  
403 GET the following object(s) :
dbVerifyStatus.0
dbVerifyError.0
Pass/Fail
(NTCIP 1201 -2.3.6, 2.3.7)
404 VERIFY that dbVerifyStatus.0 equals 'doneWithNoError' Pass/Fail
(NTCIP 1201 -2.3.1)
405 SET the following object(s) to the following values:
dbCreateTransaction.0 = 'normal'
Pass/Fail
(NTCIP 1201 -2.3.1)
406 GET the following object(s) :
phaseStartup.1
phaseStartup.2
...
Pass/Fail
(NTCIP 1202 -2.2.2.20)
407 VERIFY that phaseStartup.1 equals Phase_Startup_1 Pass/Fail
(NTCIP 1201 -2.3.1)
... ... ...

 

Slide 60:

Learning Objective #4

Database Transactions

Sample Procedure

 

Slide 61:

Learning Objective #4

Develop Sample Test Cases and Procedures Consistency Checks

 

Slide 62:

Learning Objective #4

Consistency Checks

Sample Test Case

Test Case 6.3 Title Download a database configuration with consistency errors
Description This test case verifies that the ASC properly rejects a database download that attempts to concurrently time two phases from the same ring. After initialization, it downloads several database parameters using the transaction mode and then ensures that the transaction verification results in the correct error.
The DUT must start in the standard 8-phase, dual-ring configuration.
Variables <none>
Pass/Fail Criteria The DUT shall pass every verification step included within the Test Case to pass the Test Case.

 

Slide 63:

Learning Objective #4

Consistency Checks

Sample Procedure - Initialize and Start

Step Test Procedure Results
1 PRE-CONDITION: The ASC must be configured for standard 8-phase dual-ring operation.
NOTE: This means that phaseConcurrency.1 and phaseConcurrency.2 will equal 0x0506, among other things.
 
2 GET the following object(s):
dbCreateTransaction.0
Pass/Fail
(NTCIP 1201 -2.3.1)
3 SET-UP: VERIFY that the value of dbCreateTransaction.0 is equal to 'normal' (1).  
4 SET the following object(s) to the values shown:
dbCreateTransaction.0 = 'transaction' (2)
Pass/Fail
(NTCIP 1201 -2.3.1)
5 SET the following object(s) to the values shown:
phaseConcurrency.1 = 0x020506
phaseConcurrency.2 = 0x010506
Pass/Fail
(NTCIP 1201 -2.3.1;
NTCIP 1202 -2.2.2.23)
6 GET the following object(s) :
phaseConcurrency.1
phaseConcurrency.2
Pass/Fail
(NTCIP 1202 -2.2.2.23)
7 VERIFY that phaseConcurrency.1 equals 0x0506 Pass/Fail
(NTCIP 1201 -2.3.1)

 

Slide 64:

Learning Objective #4

Consistency Checks

Sample Procedure - Download

Step Test Procedure Results
8 VERIFY that phaseConcurrency.2 equals 0x0506 Pass/Fail
(NTCIP 1201 -2.3.1)
9 SET the following object(s) to the values shown:
dbCreateTransaction.0 = 'verify' (3)
Pass/Fail
(NTCIP 1201 -2.3.1)
10 GET the following object(s) :
dbCreateTransaction.0
Pass/Fail
(NTCIP 1201 -2.3.1)
11 IF dbCreateTransaction.0 equals 'done', GOTO Step 12; otherwise GOTO Step 10  
12 GET the following object(s) :
dbVerifyStatus.0
dbVerifyError.0
Pass/Fail
(NTCIP 1201 -2.3.6, 2.3.7)
13 VERIFY that dbVerifyStatus.0 equals 'doneWithError' Pass/Fail
(NTCIP 1202 -B.1)
14 VERIFY that dbVerifyError.0 equals "PHASE 01 CONCURRENCY FAULT" or "PHASE 02 CONCURRENCY FAULT" Pass/Fail
(NTCIP 1202 - B.1)

 

Slide 65:

Learning Objective #4

Consistency Checks

Sample Procedure - Verify

Step Test Procedure Results
15 SET the following object(s) to the following values:
dbCreateTransaction.0 = 'normal'
Pass/Fail
(NTCIP 1201 -2.3.1)
16 GET the following object(s) :
phaseConcurrency.1
phaseConcurrency.2
Pass/Fail
(NTCIP 1202 -2.2.2.23)
17 VERIFY that phaseConcurrency.1 equals 0x0506 Pass/Fail
(NTCIP 1201 -2.3.1)
18 VERIFY that phaseConcurrency.2 equals 0x0506 Pass/Fail
(NTCIP 1201 -2.3.1)

 

Slide 66:

Learning Objective #4

Develop Sample Test Cases and Procedures STMP

 

Slide 67:

Learning Objective #4

STMP

Sample User Need

 

Slide 68:

Learning Objective #4

STMP

Sample Requirement

* The limitation of 13 messages is derived from NTCIP 1103 Clause 5.1.1.3.

 

Slide 69:

Learning Objective #4

STMP

Sample Dialog

A slide showing a sample STMP configuration dialog. Please see the Extended Text Description below.

(Extended Text Description: A slide showing a sample STMP configuration dialog. It is a UML sequence diagram between a "TMS" and an "ASC". It starts with the "TMS" sending a "Set(dynObjConfigStatus.x = invalid)" message and the "ASC" responding with a response message. The "TMS" then sends a "Set(dynObjConfigStatus.x = underCreation)" message and the "ASC" responds with a response message. The "TMS" then sends a "Set(dynObjConfigOwner.x, dynObjVariable.x.1, …, dynObjVariable.x.n)" message and the "ASC" responds with a response message. Finally, the "TMS" sends a "Set(dynObjConfigStatus.x = valid)" message and the "ASC" responds with a response message.)

From NTCIP 1103v02 Page 37

 

Slide 70:

Learning Objective #4

STMP

Sample Test Cases

Testing should include dynamic objects that your system plans to use plus additional dynamic objects

 

Slide 71:

Learning Objective #4

STMP

Sample Test Case

Test Case 7.1 Title Configure a dynamic object
Description This test case verifies that the ASC allows a user to configure a dynamic object. After initialization, it configures the dynamic object and verifies that the settings were saved. This test changes the definition of a dynamic object.
Variables Dynamic_Object_Number, Dynamic_Object_Owner, Object_1, Object_2, Object_3
Pass/Fail Criteria The DUT shall pass every verification step included within the Test Case to pass the Test Case.

 

Slide 72:

Learning Objective #4

STMP

Sample Procedure - Initialize and Start

Step Test Procedure Results
1 CONFIGURE: Determine the desired state of phase 1 at startup. RECORD this information as: Dynamic_Object_Number  
2 CONFIGURE: Determine the desired state of phase 1 at startup. RECORD this information as: Dynamic_Object_Owner  
3 CONFIGURE: Determine the desired state of phase 1 at startup. RECORD this information as: Object_1  
4 CONFIGURE: Determine the desired state of phase 1 at startup. RECORD this information as: Object_2  
5 CONFIGURE: Determine the desired state of phase 1 at startup. RECORD this information as: Object_3  
6 SET the following object(s):
dynObjConfigStatus.Dynamic_Object_Number = 'invalid'
Pass/Fail
(NTCIP 1103 -5.2.4)
7 SET the following object(s):
dynObjConfigStatus.Dynamic_Object_Number = 'underCreation'
Pass/Fail
(NTCIP 1103 -5.2.4)

 

Slide 73:

Learning Objective #4

STMP

Sample Procedure - Download

Step Test Procedure Results
8 SET the following object(s):
Object_1.Dynamic_Object_Number.1
Object_2.Dynamic_Object_Number.2
Object_3.Dynamic_Object_Number.3
Pass/Fail
(NTCIP 1103 -5.2.4)
9 SET the following object(s):
dynObjConfigStatus.Dynamic_Object_Number = 'valid'
Pass/Fail
(NTCIP 1103 -5.2.4)
10 GET the following object(s):
dynObjConfigStatus.Dynamic_Object_Number
Pass/Fail
(NTCIP 1103 -5.2.4)
11 VERIFY that dynObjConfigStatus.Dynamic_Object_Number equals 'valid' Pass/Fail
(NTCIP 1201 -2.3.1)

 

Slide 74:

Learning Objective #4

Develop Sample Test Cases and Procedures Load Testing

 

Slide 75:

Learning Objective #4

Load Testing

Sample Test Case / Procedure

 

Slide 76:

Learning Objective #4

Load Testing

Sample Test Procedure

 

Slide 77:

Learning Objective #4

Develop Sample Test Cases and Procedures Other Key Requirements to Consider

 

Slide 78:

Learning Objective #4

Develop Sample Test Cases and Procedures Implications for Interoperability

 

Slide 79:

Activity. A placeholder graphic with an image of hand over a computer keyboard to show that an activity is taking place.

 

Slide 80:

Learning Objective #4

Which of the following statements is true?

  1. The transaction mode must be used for all data downloads
  2. A manufacturer may not impose its own consistency checks
  3. STMP testing only needs to worry about "your" dynamic objects
  4. There is no guarantee of off-the-shelf interoperability

 

Slide 81:

Learning Objective #4

Review of Answers

A small graphical red and yellow X representing incorrect.a) Transaction mode must be used for all downloads
Incorrect; transaction mode is only required for objects designated as "P2 "parameters.

A small graphical red and yellow X representing incorrect.b) Device may not impose additional consistency checks
Incorrect; a manufacturer may define additional checks and agencies need to be aware of these.

A small graphical red and yellow X representing incorrect.c) Only test your dynamic objects
Incorrect; testing should ensure that dynamic objects will work with any future system upgrade.

A small graphical green and yellow check mark representing correct.d) No guarantee of off-the-shelf interoperability
Correct! Your specifications can impact whether off-the-shelf devices are interoperable.

 

Slide 82:

Summary of Learning Objective #4

Develop Sample Test Case Specifications and Procedures for NTCIP 1202

 

Slide 83:

Learning Objective #5 — Understand Testing Results for NTCIP 1202

 

Slide 84:

Learning Objective #5

Understand Testing Results

Sample Test Summary

See Participant Student Supplement for example

 

Slide 85:

Learning Objective #5

Understand Testing Results

Investigate Failures with Test Incident Reports

See example in Participant Student Supplement

 

Slide 86:

Learning Objective #5

Understand Testing Results

Investigate Failures with Test Incident Reports

Incident Description

 

Slide 87:

Learning Objective #5

Understand Test Results

Investigate Failures with Test Incident Reports

Incident Description

 

Slide 88:

Learning Objective #5

Understand Testing Results

Investigate Failures with Test Incident Reports

Incident Description

 

Slide 89:

Learning Objective #5

Understand Testing Results

Interpret What a Failure Means

Incident Impact

 

Slide 90:

Learning Objective #5

Understand Testing Results

Interpret What a Failure Means

Incident Description and Test Log

 

Slide 91:

Learning Objective #5

Understand Testing Results

Inspect Test Logs

Description Notes Time Result
Step 1: Time_Offset = 33   18:25:08 N/A
Step 2: GET globalTime.0   18:25:08 Passed
Step 3: RECORD globalTime.0 = 123456789   18:25:08 N/A
Step 4: SET globalTime.0 = 123456822   18:25:08 Passed
Step 5: Delay for 15 seconds   18:25:08 N/A
Step 6: Get globalTime.0   18:25:23 Passed
Step 7: Verify globalTime.0 (123456836) ~= 123456837   18:25:24 Passed

Actual test file will often provide more details about each step.

 

Slide 92:

Learning Objective #5

Understand Testing Results

Inspect Test Logs

Frame ID Type Community Error Object 1 Timestamp
1 1 Get public No Error globalTime.0 18:25:08.731
2 1 Response public No Error globalTime.0 18:25:08.785
3 2 Set public No Error globalTime.0 18:25:08.801
4 2 Response public No Error globalTime.0 18:25:08.894
5 3 Get public No Error globalTime.0 18:25:08.902
6 3 Response public No Error globalTime.0 18:25:08.999

Actual test file will often provide more details about each step.

 

Slide 93:

Learning Objective #5

Understand Testing Results

Appreciate need to repeat tests

 

Slide 94:

Learning Objective #5

Understand Testing Results

Appreciate need to repeat tests

 

Slide 95:

Activity. A placeholder graphic with an image of hand over a computer keyboard to show that an activity is taking place.

 

Slide 96:

Learning Objective #5

Which document(s) discuss potential impacts of testing failures?

  1. Test summary
  2. Test incident report
  3. Test log
  4. Test summary and test incident report

 

Slide 97:

Learning Objective #5

Review of Answers

A small graphical red and yellow X representing incorrect.a) Test Summary
Incorrect; incident reports contain an impact clause.

A small graphical red and yellow X representing incorrect.b) Test Incident Report
Incorrect; the test summary contains an evaluation.

A small graphical red and yellow X representing incorrect.c) Test Log
Incorrect; the log does not contain this information.

A small graphical green and yellow check mark representing correct.d) Test Summary and Test Incident Report
Correct! The Test Summary includes an Evaluation clause providing an overview and each Incident Report provides a detailed Impact clause.

 

Slide 98:

Summary of Learning Objective #5

Understand Testing Results for NTCIP 1202

 

Slide 99:

What We Have Learned

  1. Testing ASCs is important due to their safety- related issues.
  2. Every ASC requirement should be tested at least one time in at least one test phase using at least one method and by at least one party.
  3. In NTCIP, the test design specification, test case specification, and test procedure are typically defined in a combined test procedure table.
  4. Due to various complications there is no guarantee of off-the-shelf interoperability for ASCs.
  5. The Test Summary and Test Incident Reports reveal the impact of failures on a project.

 

Slide 100:

Resources

 

Slide 101:

Questions? A placeholder graphic image with word Questions? at the top, and an image of a lit light bulb on the lower right side.