Module 52 - T306

T306: Applying Your Test Plan to the Electrical and Lighting Management Systems Based on NTCIP 1213 ELMS Standard v03

HTML of the PowerPoint Presentation

(Note: This document has been converted from a PowerPoint presentation to 508-compliant HTML. The formatting has been adjusted for 508 compliance, but all the original text content is included, plus additional text descriptions for the images, photos and/or diagrams have been provided below.)

Slide 1:

(Extended Text Description: Welcome - Graphic image of introductory slide. A large dark blue rectangle with a wide, light grid pattern at the top half and bands of dark and lighter blue bands below. There is a white square ITS logo box with words "Standards ITS Training - Transit" in green and blue on the middle left side. The word "Welcome" in white is to the right of the logo. Under the logo box is the logo for the U.S. Department of Transportation, Office of the Assistant Secretary for Research and Technology.)

Slide 2:

(Extended Text Description: This slide, entitled "Welcome" has a photo of Ken Leonard, Director, ITS Joint Program Office, on the left hand side, with his email address, Ken.Leonard@dot.gov. A screen capture snapshot of the home webpage is found on the right hand side - for illustration only - from August 2014. Below this image is a link to the current website: www.pcb.its.dot.gov - this screen capture snapshot shows an example from the Office of the Assistant Secretary for Research and Development - Intelligent Transportation Systems Joint Program Office - ITS Professional Capacity Building Program/Advanced ITS Education. Below the main site banner, it shows the main navigation menu with the following items: About, ITS Training, Knowledge Exchange, Technology Transfer, ITS in Academics, and Media Library. Below the main navigation menu, the page shows various content of the website, including a graphic image of professionals seated in a room during a training program. A text overlay has the text Welcome to ITS Professional Capacity Building. Additional content on the page includes a box entitled What's New and a section labeled Free Training. Again, this image serves for illustration only. The current website link is: http://www.pcb.its.dot.gov.)

Slide 3:

T306: Applying Your Test Plan to the Electrical and Lighting Management Systems Based on NTCIP 1213 ELMS Standard v03

© Jim Frazer 2017

Slide 4:

Instructor

James J. Frazer

President

Gridaptive Technologies

Pompano Beach, FL, USA

Slide 5:

Learning Objectives

• Describe ELMS Testing
• Describe ELMS Test Plan Application
• Identify Relevant Elements of an ELMS Test Plan
• Describe Adaptation of a Test Plan

Slide 6:

Learning Objective 1

• Describe ELMS Testing

Slide 7:

Describe ELMS Testing

The testing life cycle, the role of test plans, and the testing to be undertaken for Electrical and Lighting Management Systems (ELMS)

• Why We Test ELMS
• Purpose of an ELMS Test Plan
• Components of an ELMS Test Plan
  • Test Design Specification
  • Test Case Specification
  • Test Procedure Specification

Slide 8:

Why We Test

To confirm that an ELMS will work as intended

The testing process provides objective evidence that the system:

• Satisfies the system requirements
• Solves the right problem
• Satisfies the user needs

Slide 9:

Why We Test

Testing and the Systems Life Cycle

(Extended Text Description: Author's relevant description: This slide has a graphic of a blue large letter "V" which is a graphical representation of systems engineering process. Systems life cycle diagram shows testing to be undertaken at various stages, as well as the types of testing to be utilized. In this case, System Validation, System Verification and Deployment, Subsystem Verification and Unit/Device Testing are highlighted in red. A very detailed explanation of how a Vee diagram is used in PCB modules is provided below. This level of detail is not necessary and may be skipped. Explanation of V Diagram A graphic of the systems engineering process (SEP) is shown. The main graphic of the SEP is a V-shaped diagram with some additional horizontal "wings" on the left and right side of the top of the V. Starting from the left "wing" the steps are regional architecture, needs assessment, concept selection, project planning, and systems engineering management planning. At this point the steps begin to descend the left side of the V with concept of operations, system requirements, high-level design, detailed design, and software hardware and field installation. At this point the steps begin to ascend the right side of the V with unit device testing, subsystem verification, system verification and deployment, system validation, and operations and maintenance. At this point the steps are on the right "wing" of the V with changes and upgrades and retirement/replacement. The following arrows supplement the figure: 1. A time line at the bottom of the figure indicating that all steps proceed over time with the steps forming the V overlapping one another. 2. A downward arrow on the left side of the V that indicates that these steps deal with decomposition and definition. 3. An upward arrow on the right side of the V that indicates that these steps deal with integration. 4. A horizontal arrow near the bottom of the center of the V that connects the detailed design step to the unit testing step. 5. A horizontal arrow a little higher that connects the subsystem requirements step with the subsystem verification step. 6. A horizontal arrow a little higher that connects the system requirements step with the system verification step. 7. A horizontal arrow near the top of the V that connects the concept of operations step with the system validation step. Finally, major steps are separated by a barrier that is labeled "document approval." This graphical view of SEP life cycle process used to show where SSM user needs/requirements are located. In this slide, at the top of the Vee, there are three boxes in yellow.)

Slide 10:

Why We Test

To confirm that an ELMS system will work as intended

(Extended Text Description: Author's relevant description: This slide has a graphic of a blue large letter "V" which is a graphical representation of systems engineering process. Focus upon the traceability of testing to the underlying requirements and foundational user needs. Systems life cycle diagram shows testing to be undertaken at various stages, which includes Test Planning, Testing Documentation (Test Plan), and Testing Process. Note that test planning is done earlier than test design documents. Further description of the Vee diagram is found in previous slide 9.)

Slide 11:

Purpose of a Test Plan

Does the system conform to the requirements?

• A test plan is a document describing:
  • Scope (technical management)
  • Approach
  • Resources needed
  • Schedule to complete
• A test plan identifies:
  • Test items
  • Features to be tested
  • Testing tasks
  • Risks requiring contingency plan

Testing determines whether the system conforms to the requirements and whether it satisfies its intended use and user needs (IEEE-829-2008).

Slide 12:

Components of a Test Plan

Relationship between Components

(Extended Text Description: Author's relevant description: This slide has a graphic of the hierarchical relationship between test plan components. A test plan provides a description of the overall approach to testing all of the requirements to be verified. The Test Plan outlines the scope, approach, resources, and schedule of testing activities. Breakdown of Key Point: This first key point describes a test plan – the master document that will include the test cases. Explaining this key point shows the hierarchical structure (test plan – test design specification – test case – test procedures) that is required to develop a fully system-engineered test plan. To the left of the diagram are bullet points with arrows corresponding to the related sections of the diagram:

• Test Plan Specification
• Test Design Specification
• Test Case Specification
• Test Procedure Specification

Image Courtesy of ITS PCB Course T321, Slide 92.)

Slide 13:

Components of a Test Plan

Test Plan Specification

• Test plan specifications detail objectives, target market, internal beta team, and processes for a specific test for a software or hardware product
• Test plan specifications contain a detailed understanding of the comprehensive workflow

Slide 14:

Components of a Test Plan

Test Design Specification

• Test design specifications result in a collection of test cases that are intended to be used to test a specified set (test suite) of behavior
• A test suite contains detailed instructions or goals for each collection of test cases and information on the system configuration to be used during testing

Slide 15:

Components of a Test Plan

Test Case Specification (TCS)

A test case specification is a set of conditions under which a tester will determine whether the system is working as it was originally intended to do

Slide 16:

Components of a Test Plan

Test Procedure Specification

• Test procedure specification defines a process that produces a test result
• It is a technical operation that consists of determining the characteristics of a given product, process, or service according to a specified procedure

Slide 17:

Slide 18:

Question

Which is not a component of an ELMS test plan?

Answer Choices

1. Test Facilitation
2. Test Design Specification
3. Test Case Specification
4. Test Procedure Specification

Slide 19:

Review of Answers

a) Test Facilitation
Correct! Test facilitation is not part of an ELMS test plan.

b) Test Design Specification
Incorrect. Test Design Specification is part of an ELMS test plan.

c) Test Case Specification
Incorrect. Test Case Specification is part of an ELMS test plan.

d) Test Procedure Specification
Incorrect. Test Procedure Specification is part of an ELMS test plan.

Slide 20:

Learning Objectives

• Describe ELMS Testing
• Describe ELMS Test Plan Application

Slide 21:

Learning Objective 2

• Describe ELMS Test Plan Application

Slide 22:

ELMS Test Plan Application

Steps in Developing an ELMS Test Plan

• Identify requirements to be tested/not to be tested for each testing phase
• Identify test methodology
• Introduce and describe the Requirements to Test Case Traceability Matrix (RTCTM)
• Plan logistics of testing
• Estimate level of effort for testing
• Evaluate risks
• Plan project closeout

Slide 23:

ELMS Test Plan Application

Develop a Sample Test Plan

Identify Requirements to Test:

• Requirements are found in the Protocol Requirements List (PRL)
• Module A306b identified how to define ELMS requirements
  • See Participant Student Supplement for list of sample requirements
• Every requirement should be tested:
  • During at least one test phase
  • Using at least one method
  • By at least one party
• Extent of agency testing is a risk management issue

Slide 24:

ELMS Test Plan Application

Develop a Sample Test Plan

Identify Test Plan Level

• Each test level will have its own test plan ° Prototype
  • Design Approval
  • Factory Acceptance
  • Incoming Device
  • Site Acceptance
  • Burn-in
• Often further divided
  • NTCIP testing
  • Hardware testing
  • Etc.

Slide 25:

ELMS Test Plan Application

Develop a Sample Test Plan Approach

• Identify Test Methodology
• Inspection
• Analysis
• Demonstration
• Formal testing
• Consider testing scenarios
  • Positive test(s)
  • Negative test(s)
  • Boundary test(s)

Slide 26:

ELMS Test Plan Application

Develop a Sample Test Plan

Requirements to Test Case Traceability Matrix (RTCTM)

Requirement ID	Requirement	Test Case ID	Test Case
3.5.4.1.1.1	Retrieve Luminaire Pole Identifier	3.5.4.1.1	Retrieve Luminaire Pole Identifier
3.5.4.1.1.2	Retrieve Luminaire Location	3.5.4.1.1.2	Retrieve Luminaire Location
3.5.4.1.3	Configure Luminaire Mode	3.5.4.1.3.1	Configure Luminaire Mode
		3.5.4.1.3.2	Incorrectly Configure Luminaire Mode
3.5.4.1.4.1	Configure Luminaire Color Temperature	3.5.4.1.4.1.1	Configure Luminaire Color Temperature
		3.5.4.1.4.1.2	Incorrectly Configure Luminaire Color Temperature

Slide 27:

ELMS Test Plan Application

Develop a Sample Test Plan

Identify the Test Environment

(Extended Text Description: Author's relevant description: This slide includes an image of a subset of a device under test communication with a test application. A data analyzer is pictured "listening" to the communications stream. Focus upon: The test environment needs to be defined in terms of hardware and software required.)

Source: NTCIP 8007, page 13

Slide 28:

ELMS Test Plan Application

Develop a Sample Test Plan

Plan Logistics of Testing

• Where will tests be performed?
  • Safety during on-site testing
• Who is responsible for what?
  • Power
  • Tools
  • Tables
  • Protection from elements
  • Local assistance for remote testing
• What happens if testing is suspended?

Slide 29:

ELMS Test Plan Application

Develop a Sample Test Plan

Estimate effort, schedule, and budget for:

• Preparing test plan
• Preparing test cases
• Preparing test procedures
• Performing multiple rounds of testing
  • Performing tests
  • Investigating problems
  • Preparing test documentation

Slide 30:

ELMS Test Plan Application

Develop a Sample Test Plan

Understanding the Impact of a Failure

Requirement ID	Requirement	Test Case ID	Test Case
3.5.4.1.1.1	Retrieve Luminaire Pole Identifier	3.5.4.1.1	Retrieve Luminaire Pole Identifier
3.5.4.1.1.2	Retrieve Luminaire Location	3.5.4.1.1.2	Retrieve Luminaire Location
3.5.4.1.3	Configure Luminaire Mode	3.5.4.1.3.1	Configure Luminaire Mode
		3.5.4.1.3.2	Incorrectly Configure Luminaire Mode
3.5.4.1.4.1	Configure Luminaire Color Temperature	3.5.4.1.4.1.1	Configure Luminaire Color Temperature
		3.5.4.1.4.1.2	Incorrectly Configure Luminaire Color Temperature

Slide 31:

ELMS Test Plan Application

Develop a Sample Test Plan

Understanding the Impact of a Failure

User Need ID	User Need	Requirement ID	Requirement
2.5.2.1.1.1	Retrieve Luminaire Information
		3.5.4.1.1.1	Retrieve Luminaire Pole Identifier
		3.5.4.1.1.2	Retrieve Luminaire Location
		3.5.4.1.1.3	Retrieve Luminaire Mode
		3.5.4.1.1.4	Retrieve Luminaire Zone
		3.5.4.1.1.5	Retrieve Luminaire Vendor Information
		3.5.4.1.1.6	Retrieve Luminaire Light Source Type
		3.5.4.1.1.7	Retrieve Luminaire Wattage
		3.5.4.1.1.8	Retrieve Luminaire Voltage

Slide 32:

ELMS Test Plan Application

Develop a Sample Test Plan

Plan Project Closeout

• Have a plan
• Understand the impacts of accepting a failure

Slide 33:

Slide 34:

Question

Which of the following ELMS statements is false?

Answer Choices

1. Every ELMS requirement should be tested
2. You should only need to perform your ELMS test plan once
3. Some ELMS testing may be performed by the manufacturer
4. ELMS traceability tables can help you assess the impact of a test failure

Slide 35:

Review of Answers

a) Every ELMS requirement should be tested
True. Every requirement should be tested.

b) You should only need to perform your ELMS test plan once
False (correct). This statement is not true. Testing will often reveal problems; these should be fixed and the device retested.

c) Some testing may be performed by the manufacturer
True. Testing may be performed by the agency, the manufacturer, or a third party.

d) ELMS traceability tables can help you assess the impact of a test failure
True. Traceability tables allow you to identify the user needs that will not be completely fulfilled.

Slide 36:

Learning Objectives

• Describe ELMS Testing
• Describe ELMS Test Plan Application
• Identify Relevant Elements of an ELMS Test Plan

Slide 37:

Learning Objective 3

• Identify Relevant Elements of an ELMS Test Plan

Slide 38:

What Is Being Tested?

Only Project-Specific Requirements Are Tested

(Extended Text Description: This slide contains the following table with values circled in red in the Support column: rows 2, 4, 6, 7: Yes, rows 5 and 8: No:

User Need ID	User Need	FR ID	Functional Requirement	Conformance	Support	Additional Specifications
2.5.2.2.2	Control Electrical Service			0	Yes / No
		3.5.5.2.1	Control Electrical Service by Permanent/Continuous Override	M	Yes
		3.5.5.2.2	Control Electrical Service by Transitory Override	0	Yes / No
		3.5.5.2.3	Control Electrical Service by Timed Override	0	Yes / No
		3.5.5.2.4	Control Electrical Service in Stagger Mode	0	Yes / No
		3.5.5.2.5	Control Electrical Service by Photocell	0	Yes / No
		3.5.5.2.6	Control Electrical Service by Adaptive Means	0	Yes / No

)

Slide 39:

Test Design, Test Cases, and Test Procedures

Designing Test Case Specifications and Procedures

• Review guidance from IEEE 829-2008 and NTCIP 8007
• Apply guidance to sample dialog
• Key Differences Between the Two Approaches
  • IEEE standard approach is applicable to all ITS standards including C2C and C2F
  • IEEE standard approach separates test cases from test procedures, while previous efforts combined both, such as per NTCIP 8007 information report
  • IEEE standard approach allows reuse of test procedures, where agencies typically place more efforts
  • IEEE standard approach includes a test plan and method to split testing into test designs, and includes test reports

Slide 40:

Test Design, Test Cases, and Test Procedures

Designing Test Case Specifications and Procedures

What Does IEEE 829-2008 Provide?

• Test Plan
• Test Design Specification
• Test Case Specification
• Test Procedure Specification
• Test Reports
  • Test Logs
  • Test Anomaly Report
  • Test Report
• Testing professionals across ITS are familiar with these definitions/formats

Slide 41:

Test Design, Test Cases, and Test Procedures

Designing Test Case Specifications and Procedures

NTCIP 8007 Components

• NTCIP 8007 describes how these can be combined for NTCIP testing
  • Test case identifier
  • Purpose
  • Inputs
  • Pass/Fail criteria
  • Procedure steps
    • Can reference other often used procedures
    • Expected outputs
    • Features tested
• Defines terms that can be used in test steps for NTCIP testing

Slide 42:

Test Design, Test Cases, and Test Procedures

Designing Test Case Specifications and Procedures

Sample Basic Dialog: Set Time Dialog

(Extended Text Description: Author's relevant description: This slide has an image of a communications message between a management station and an ELMS device. The slide is an example of a simple dialog between a Management Station and an ELMS device. A comprehensive list of dialogs is included in the standard. Test Case also seeks to test boundaries with Data objects variable ranges; that part is missing. Dialogs are tested for sequence of message content.)

Slide 43:

Test Design, Test Cases, and Test Procedures

Designing Test Case Specifications and Procedures

• Specify Test Case First
  • "What You Are Testing"
• Then Specify Test Procedure
  • "How You Run The Test"

Slide 44:

Test Design, Test Cases, and Test Procedures

Designing Test Case Specifications

Specify Each Test Case

Requirement ID	Requirement	Test Case ID	Test Case
2.2.1	Set Time	2.2.1	Set Time

Test Case
Test Case 2.2.1	Test Case Name	Set Time
	Description	This test case verifies that the ELMS properly tracks time. It advances the clock by a user-defined amount, waits a few seconds, retrieves the time, and verifies it indicates an appropriate value.
	Variable	GlobalTime as defined in NTCIP 1213 V3.0
	Pass/Fail Criteria	The DUT shall pass every verification step included within the Test Case to pass the Test Case.

Slide 45:

Test Design, Test Cases, and Test Procedures

Designing Test Case Procedures

• Data exchanges should follow defined dialogs
• Return the device to its original state (generally)
• Verification steps should cite the relevant requirement
• A test case typically tests multiple requirements
• NTCIP 8007 precisely defines standardized step types
• A "SET" operation includes nine specific verification checks related to the Simple Network Management Protocol (SNMP) response packet

Slide 46:

Test Design, Test Cases, and Test Procedures

Designing Test Case Procedures

Steps of a Sample Procedure

Step Number	Test Procedure	Results
1	CONFIGURE: Determine the number of seconds to advance the clock in the ELMS
2	GET the following object(s): globalTime.0	Pass/Fail
3	RECORD the RESPONSE VALUE for globalTime.0 as Start_Time
4	SET the following object(s): globalTime.0 = Start_Time + Time_Offset	Pass/Fail
5	DELAY for 15 seconds
6	GET the following object(s): globalTime.0	Pass/Fail
7	VERIFY that the RESPONSE VALUE for globalTime.0 is roughly equal to Start_Time + Time_Offset + 15	Pass/Fail

Slide 47:

Adapting the Test Plan

The process of adapting the test plan based on selected user needs and requirements

• We have described the components of a test plan
• We have examined the major components of test cases and test procedures in detail
• Next we will create a project-specific ELMS test plan

Slide 48:

Slide 49:

Question

Where can you find definitions for terms that can be used in NTCIP test steps?

Answer Choices

1. IEEE 829
2. NTCIP 8007
3. ISO 9001
4. Student Supplement

Slide 50:

Review of Answers

a) IEEE 829
Incorrect. IEEE 829 defines sample outlines for test documentation, but does not define steps for NTCIP.

b) NTCIP 8007
Correct! NTCIP 8007 defines a number of terms that can be used in test steps for NTCIP testing.

c) ISO 9001
Incorrect. ISO 9001 deals with quality management, but does not deal directly with NTCIP testing.

d) Student Supplement
Incorrect. The student supplement provides samples of test procedures, but it does not define the test terms.

Slide 51:

Learning Objectives

• Describe ELMS Testing
• Describe ELMS Test Plan Application
• Identify Relevant Elements of an ELMS Test Plan
• Describe Adaptation of a Test Plan

Slide 52:

Learning Objective 4

• Describe Adaptation of a Test Plan

Slide 53:

Develop an NTCIP 1213 v03 Test Design Specification

Background

Information Sources:

NTCIP 1213 v03 - National Transportation Communications for ITS Protocol Object Definitions for Electrical and Lighting Management Systems

• Protocol Requirements List (PRL)
• Requirements Traceability Matrix (RTM)
• Testing documentation
  • Required but not supplied in standard
  • Must be created

Slide 54:

Develop an NTCIP 1213 v03 Test Design Specification

Background

Characteristics of the NTCIP 1213 v3.0 (ELMS) Standard

• ELMS is a Center-to-Field Communications Standard
• ELMS contains System Engineering (SE) content (the standard has a PRL and an RTM)
• ELMS does not contain Test Procedures

Slide 55:

Develop an NTCIP 1213 v03 Test Design Specification

Background

Protocol Requirements List (PRL)

• Contains user needs
• Contains functional requirements
• Describes relationship between needs and requirements
• Project-specific requirements are identified by project-level Protocol Requirements List (PRL)

Slide 56:

Develop an NTCIP 1213 v03 Test Design Specification

Background

Requirements Traceability Matrix (RTM)

• Contains functional requirements
• Contains object dialogs
• Describes relationship between requirements and object dialogs
• A project-specific Requirements Traceability Matrix (RTM) references relevant design content needed to define the inputs and outputs for the test case specification

Slide 57:

Develop an NTCIP 1213 v03 Test Design Specification

Background

Context Diagram

(Extended Text Description: Author's relevant description: This slide has an image of a communications message between a management station and an ELMS device. This is an example of the electricalserviceSwitchState dialog. Each dialog includes a numeric object identifier ID as well as an object name. Each project-specific object will need to be examined.)

Slide 58:

NTCIP 1213 v03 Test Design Specification

Testing Documentation

Step 1: Select Your User Needs in the PRL

Slide 59:

NTCIP 1213 v03 Test Design Specification

Testing Documentation

Step 2: Use Project RTM to Identify Objects and Dialogs to Be Tested

Requirements Traceability Matrix (RTM)
FR ID	Functional Requirement	Dialog ID	Object ID	Object Name	Additional Specifications
3.5.5.2.2	Control Electrical Service by Transitory Override	G.3
			5.5.1.6	electricalserviceSwitchMode
3.5.5.2.3	Control Electrical Service by Timed Override	4.2.13
			5.5.1.6	electricalserviceSwitchMode
			5.5.1.7	electricalServiceSwitchModeTime
3.5.5.2.4	Control Electrical Service in Stagger Mode	G.3
			5.5.1.28	electricalserviceSwitchState
3.5.5.2.5	Control Electrical Service by Photocell	G.3
			5.5.1.29	electricalservicePhotocellIndex
3.5.5.2.6	Control Electrical Service by Adaptive Means	G.3
			5.5.1.6	eletricalserviceSwitchMode

Slide 60:

NTCIP 1213 v03 Test Design Specification

Testing Documentation

Step 3: Develop Test Case Objective

(Extended Text Description: Author's relevant description: This slide includes an image of a test case objective. In this example we have defined one objective addressing four requirements. Notice that the test case verifies that the data value of the OBJECTS requested are within specified ranges. The key text of the image includes:

To verify system interface implements (positive test case) requirements for a series of object requests for:

3.5.5.2.2 electricalserviceSwitchMode

3.5.5.2.3 electricalserviceSwitchModeTime

3.5.5.2.4 electricalserviceSwitchState

3.5.5.2.5 electricalservicePhotocellIndex

The test case verifies that the data value of the OBJECTS requested are within specified ranges. The object identifier (OID) of each object requested is the only input required. An output specification is provided to show valid value constraints per the NTCIP 1205 v01 object definitions.)

Slide 61:

NTCIP 1213 v03 Test Design Specification

Testing Documentation

Step 3: Develop Test Case Objective (continued)

Test Case Output Specification
ID:TCOS001		Title: Status Condition within the Device
Data Concept ID	Data Concept Name (Variable)	Data Concept Type	Value Constraints
3.5.5.2.2	elertrcialserviceSwitchMode	Data Element
3.5.5.2.3	electrcialserviceSwitchModeTime	Data Element
3.5.5.2.4	electrcialserviceSwitchModeState	Data Element
3.5.5.2.5	electrcialservicePhotocellIndex	Data Element

Slide 62:

NTCIP 1213 v03 Test Design Specification

Testing Documentation

Step 4: Identify Dialogs, Inputs, Outputs

(Extended Text Description: Author's relevant description: This slide includes an image of a test case output specification table. This is a definition from the standard. The key is to notice how Type and Valid Range are defined here as octet string and a range of 1 to 9.)

Slide 63:

NTCIP 1213 v03 Test Design Specification

Testing Documentation

Step 4: Identify Dialogs, Inputs, Outputs (continued)

(Extended Text Description: Author's relevant description: This slide includes an image of a test case output specification table. This is another definition from the standard. The key is to notice how Type and Valid Range are defined here as integer and a range of 0 to 65535.)

Slide 64:

NTCIP 1213 v03 Test Design Specification

Testing Documentation

Step 5: Document Value Constraints for Inputs

(Extended Text Description: Author's relevant description: This slide includes an image of a test case output specification table. From the standard, we enter the value constraints into the Test Case Input Specification. The Value Constraints are highlighted in red in the table below:

Test Case Input Specification
ID TCI201		Title: Input Specification for electricalserviceSwitchMode (Positive test case)
Data Concept ID	Data Concept Name (Variable)	Data Concept Type	Value Constraints
3.5.5.52.2	electricalserviceSwitchMode	Data Element	1 = "permanentOn" 2 = "permanentOff" 3 = "schedule" 4 = "transitoryOn" 5 = "transitoryOff" 6 = "timedOn" 7 = "timedOff" 8 = "none" 9 = "adaptive"

)

Slide 65:

NTCIP 1213 v03 Test Design Specification

Testing Documentation

Step 5: Document Value Constraints for Outputs

(Extended Text Description: Author's relevant description: This slide includes an image of a test case output specification table. We also enter the value constraints into the Test Case Output Specification. At this point it is important to remember that negative testing is also important. The Value Constraints are highlighted in red in the table below:

Test Case Output Specification
ID TCI201		Title: Output Specification for electricalserviceSwitchMode (Positive test case)
Data Concept ID	Data Concept Name (Variable)	Data Concept Type	Value Constraints
3.5.5.52.2	electricalserviceSwitchMode	Data Element	1 = "permanentOn" 2 = "permanentOff" 3 = "schedule" 4 = "transitoryOn" 5 = "transitoryOff" 6 = "timedOn" 7 = "timedOff" 8 = "none" 9 = "adaptive"

)

Slide 66:

NTCIP 1213 v03 Test Design Specification

Testing Documentation

Step 6: Complete Test Case

(Extended Text Description: Author's relevant description: This slide includes an image of a test case output specification table. In this slide we are completing a simplified test case. Notice the additional items of Environmental Needs, Tester/Reviewer, Special Procedure Requirements, and Intercase Dependencies. Those areas are highlighted in red in the table below:

Test Case
ID TCI201	Title: electricalserviceSwitchMode Dialog Verification (Positive Test Case)
Objective	To verify system interface implements(positive test case) requirements for object: electricalserviceSwitchMode
Inputs	Use valid inputs as defined by test case input specification
Outcomes	All data are returned and verified as correct: correct sequence of message exchanges, structure of data, and valid value of data content. See Test Case Output Specification for details.
Environmental Needs:	No additional needs outside of those specified in the test plan.
Tester/Reviewer	JF
Special Procedure Requirements	None
Intercase Dependencies	None

)

Slide 67:

Developing Test Cases and Procedures for Extensions

Supporting Objects Not in the Standard

Extending the Standard complicates interoperability and interchangeability

• Not achievable unless all design details are known
• Extensions are relatively custom solutions, resulting in:
  • Increased specification costs
  • Increased development costs
  • Increased testing costs
  • Increased integration costs
  • Longer deployment timeframe
  • Increased maintenance costs

Slide 68:

Developing Test Cases and Procedures for Extensions

Supporting Objects Not in the Standard

Extensions should only be considered when:

• NTCIP features are inadequate to meet needs
• Benefits of extension outweigh added costs

Slide 69:

Developing Test Cases and Procedures for Extensions

Supporting Objects Not in the Standard

Extended equipment should be designed to:

• Appropriately integrate with NTCIP-only deployments
• Minimize added complexity

Slide 70:

Developing Test Cases and Procedures for Extensions

If You Do Choose to Test Objects Not in the Standard

• Adhere to the relationships between the PRL, RTM, and RTCTM, as well as the underlying user needs and measurable functional requirements
• The main purpose of Test Design is to identify the features to be tested by a particular level test (e.g., unit test)
• The features to be tested are included in the RTCTM
  • Based on a Requirements Traceability Matrix (RTM)

Slide 71:

Test Procedure Generator Tool (TPG)

Introduction to the Test Procedure Generator

• What is the TPG?
• Why is the TPG important?
• What are the benefits?
• How do you use the TPG?
• How does it fit into testing for NTCIP Standards?
• Where does a user obtain the TPG?

Slide 72:

Test Procedure Generator Tool (TPG)

What Is the TPG and How Does It Work?

• USDOT has released the version 2.1 of the TPG tool for the ITS Standards communities
• TPG v2.1 supports development and deployment NTCIP Center-to-Field (C2F) Device Interface Standards with Systems Engineering Content
• TPG is a Windows-based software tool that uses Microsoft Word to input the NTCIP Standards and output Test Procedures
• TPG supports ITS Standard developers as well as deployers (local and state agencies) of NTCIP C2F Standards

Slide 73:

Test Procedure Generator Tool (TPG)

What Is the TPG and How Does It Work?

• For deployers and local agencies, the TPG guides the development of test procedures by: Loading and processing the standard to be implemented including the requirements, dialogs, and objects
• Basing the Test Procedures on the user-selected requirements in NTCIP C2F Standard

Slide 74:

Test Procedure Generator Tool (TPG)

What Is the TPG and How Does It Work?

• Uses standardized and consistent language for Test Procedures development, including:
  • Standard keywords, variables, and object names imported directly from the standard
• Outputs an XML file that can be consistently interpreted by vendors and testing staff for their test suites
• Standards Deployers can use the TPG to create consistent Test Procedures
• Remember: The TPG is not a testing tool!

Slide 75:

Test Procedure Generator Tool (TPG)

Benefits of the TPG

• Agencies can use the TPG to develop consistent test procedures for verifying conformance and compliance
• Using the TPG tool will reduce developmental risks, effort, and the cost of developing standards and test procedures

Slide 76:

Test Procedure Generator Tool (TPG)

(Extended Text Description: Author's relevant description: This slide includes an image that describes the focus of the TPG as test cases and test procedures. As described previously, the Test Program Generator Tool (TPG) automates steps 3 and 4 "Test Cases" and Test Procedures" of the test planning process. It does not address the Test Plan, Test Design Specifications, Test Execution, or Test Reports.)

Slide 77:

Test Procedure Generator Tool (TPG)

Role of the TPG in Testing

• Supports off-the-shelf interoperability
• Promotes the systems engineering process by giving users support in creating test procedures
• Standardized and easily available Test Procedures that are conformant to the standard help to eliminate the proprietary system elements

Slide 78:

Test Procedure Generator Tool (TPG)

Using the TPG - Start a New Session

(Extended Text Description: Author's relevant description: This slide includes an image of TPG software splash screen. The first step in using the TPG is to start a new session by loading the NTCIP 1213 v03 standard in Word doc 2010 format. The New Session dialog shows several options, including: NTCIP C2F Device Interlace Standard Number, NTCIP Standard Major Version Number, NTCIP Standard Minor Version Number, NTCIP Standard Revision Letter (Optional).)

Slide 79:

Test Procedure Generator Tool (TPG)

Using the TPG - The Graphical User Interface

(Extended Text Description: Author's relevant description: This slide includes an image of TPG software user interface. This slide represents the graphical user interface of the TPG Tool. A tree view (the session panel) is on the left. Session status is in a ribbon at the bottom. The interface shows TPG Menu Items, Document Tabs, Embedded Microsoft Word 2010 Document Menu Items, Embedded Microsoft Word 2010 Document, NTCIP 1213 version v03, TPG Session Status, TPG Command Status.)

Slide 80:

Test Procedure Generator Tool (TPG)

Using the TPG - Create a New Test Procedure

(Extended Text Description: Author's relevant description: This slide includes an image of TPG software "create a new test procedure" interface. The next step (Step 2) is to create a new Test Procedure.

A Current Test Procedure windows displays the following example table:

Test Procedure: 01.00	Select the Test Procedure->Define Header Menu Item to enter the Test Procedure Title
Description:	Select the Test Procedure->Define Header Menu Item to enter the Test Procedure Description
Require me nt(s):	Select the Test Procedure->Select Requirements Menu Item to enter the Test Procedure Requirements
Variable(s):	Select the Test Procedure->Define Variables menu item to enter the Test Procedure Variables
Pass/Fail Criteria:	Select the Test Procedure->Define Header Menu Item to enter the Test Procedure Pass/Fail Criteria

)

Slide 81:

Test Procedure Generator Tool (TPG)

Using the TPG - Create a New Test Procedure

(Extended Text Description: Author's relevant description: This slide includes an image of TPG software "create a new test procedure" interface. Creating a new Test Procedure begins with defining a Test Procedure Title and Description.)

Slide 82:

Test Procedure Generator Tool (TPG)

Using the TPG - Select Your Requirements

(Extended Text Description: Author's relevant description: This slide includes an image of TPG software "select new requirements" interface. Step 3 is to "Select Your Requirements" by checking the checkboxes of project-specific requirements. Some example requirements include: 3.4.1.1 Retrieve Data, 3.4.1.2 Deliver Data, 3.4.1.3 Explore Data, 3.4.2.1 Determine Current Configuration of Logging Service.)

Slide 83:

Test Procedure Generator Tool (TPG)

Using the TPG - The Test Procedure

(Extended Text Description: Author's relevant description: This slide includes an image of TPG software "completed test procedure" interface. This slide represents the Test Procedure as defined so far. Example content includes: Current Test Procedure, Test Procedure: 01.00, Determine Sign Type and Technology, Description: This test case verifies that the DMS indicates that it is the sign type and uses the technology as required by the specification.)

Slide 84:

Test Procedure Generator Tool (TPG)

Using the TPG - Create Your Variables

(Extended Text Description: Author's relevant description: This slide includes an image of TPG software "create your variables" interface. Step 4 is to select your variable objects. Example Create a New Variable dialog has example variables: Select Test Procedure Variable(s): Required_Sign_Technology [eventClassClearTime], Required_Sign_Type [INTEGER].)

Slide 85:

Test Procedure Generator Tool (TPG)

Using the TPG - Create Your Test Procedure Step

(Extended Text Description: Author's relevant description: This slide includes an image of TPG software "test your procedure step" interface. Step 5 is to create your Test Procedure Step. Test Procedure Step dialog is shown.)

Slide 86:

Test Procedure Generator Tool (TPG)

Using the TPG - Test Procedure Results

(Extended Text Description: Author's relevant description: This slide includes an image of TPG software "your procedure test results" interface. This slide represents your project-specific Test Procedure results. Example Set of Test Procedures, Pass/Fail Criteria: The device under test (DUT) shall pass every verification step included within the Test Case in order to pass the Test Case. Example columns show Test Step Number, Test Procedure, Results.)

Slide 87:

Test Procedure Generator Tool (TPG)

How to Obtain the TPG

• TPG v2.1 updates include: Compatibility with Windows 7 Professional
• Compatibility with Microsoft Office 2010
• For more information and to acquire the TPG, please visit:
  https://www.standards.its.dot.gov/DeploymentResources/Tools
• The free download package includes:
  • TPG v2.1 Installation file
  • TPG User Manual
• TPGSupport@noblis.org

Slide 88:

Slide 89:

Question

Which of the following statements is false?

Answer Choices

1. TPG v2.1 supports development and deployment NTCIP Center-to-Field (C2F) Device Interface Standards with Systems Engineering Content
2. TPG is a testing tool
3. TPG is a Windows-based software tool that uses Microsoft Word to input the NTCIP Standards and output Test Procedures
4. TPG supports ITS Standard developers as well as deployers (local and state agencies) of NTCIP C2F Standards

Slide 90:

Review of Answers

a) TPG v2.1 supports development and deployment NTCIP Center-to-Field (C2F) Device Interface Standards with Systems Engineering Content
Incorrect. TPG does support development and deployment NTCIP Center-to-Field (C2F) Device Interface.

b) TPG is a testing tool
Correct! False, TPG is not a testing tool.

c) TPG is a Windows-based software tool that uses Microsoft Word to input the NTCIP Standards and output Test Procedures
Incorrect. TPG is a Windows-based software tool.

d) TPG supports ITS Standard developers as well as deployers (local and state agencies) of NTCIP C2F Standards.
Incorrect. TPG supports ITS Standard developers as well as deployers.

Slide 91:

Module Summary

• Describe ELMS Testing
• Describe ELMS Test Plan Application
• Identify Relevant Elements of an ELMS Test Plan
• Describe Adaptation of a Test Plan

Slide 92:

We Have Now Completed the ELMS Curriculum

Module A306a:
Understanding user needs for Electrical and Lighting Management Systems Based on NTCIP 1213 v03

Module A306b:
Specifying requirements for Electrical and Lighting Management Systems Based on NTCIP 1213 v03

Module T306:
Applying Your Test Plan to the Electrical and Lighting Management Systems Based on NTCIP 1213 v03

Slide 93:

Thank you for completing this module.

Feedback

Please use the Feedback link below to provide us with your thoughts and comments about the value of the training.

Thank you!