Module 18 - T313

T313: Applying Your Test Plan to the NTCIP 1204 v03 ESS Standard

HTML of the Course Transcript

(Note: This document has been converted from the transcript to 508-compliant HTML. The formatting has been adjusted for 508 compliance, but all the original text content is included.)


Shelley Row
ITS Standards can make your life easier. Your procurements will go more smoothly and you'll encourage competition, but only if you know how to write them into your specifications and test for them. This module is one in a series that covers practical applications for acquiring and testing standards-based ITS systems. I am Shelley Row the director of the ITS Joint Program Office for USDOT and I want to welcome you to our newly redesigned ITS standards training program of which this module is a part. We are pleased to be working with our partner, the Institute of Transportation Engineers, to deliver this new approach to training that combines web based modules with instructor interaction to bring the latest in ITS learning to busy professionals like yourself. This combined approach allows interested professionals to schedule training at your convenience, without the need to travel. After you complete this training, we hope that you will tell colleagues and customers about the latest ITS standards and encourage them to take advantage of the archived version of the webinars. ITS Standards training is one of the first offerings of our updated Professional Capacity Training Program. Through the PCB program we prepare professionals to adopt proven and emerging ITS technologies that will make surface transportation safer, smarter and greener which improves livability for us all. You can find information on additional modules and training programs on our web site www.pcb.its.dot.gov. Please help us make even more improvements to our training modules through the evaluation process. We look forward to hearing your comments. Thank you again for participating and we hope you find this module to be helpful.

Nicola Tavares
Today’s module is T313, Applying Your Test Plans to the NTCIP 1204 version 3 ESS standard. The target audience includes engineering staff, operations and maintenance staff, traffic management staff, testing staff, includes testing personnel and systems integrators with specialized capabilities and public and private sector staff. Your instructor is Russ Brookshire. He is the product manager for embedded systems for intelligent devices. He has been involved in manufacturing and testing ITS products for more than 20 years. These included the design and testing of several ESS systems that integrated multiple sensors along with a snow plow monitoring system that included pavement treatment systems. Most recently he has worked on developing the test procedures for NTCIP 1203 version 3 dynamic message sign.

Russ Brookshire
Hi. Thank you very much, Nicola and welcome everyone to this webinar, T313 Applying Your Test Plan to the NTCIP 1204 version 3 ESS Standard. Now, there are a few recommended prerequisites for this webinar, starting with T101, an introduction to ITS Standards testing which the next would be T201, how to write a test plan which goes over how to actually create a test plan given the information that’s in some of the standards and also specifications that are generated by the agency. T202, an overview of test design specifications, test cases and test procedures. This goes into creating some of the other test documents that formed the entire test plan. And we’ll be going over some of these in this webinar and particularly how they apply to environmental sensor stations. In addition, A313a Understanding User Needs for ESS Systems based on the NTCIP 1204 version 3 standard and also A313b Specifying Requirements for ESS Systems based on the NTCIP 1204 version 3 standard. Note that the first three of these are part of the testing curriculum path in the webinars. And the last two A313a and A313b are part of the acquisition curriculum path.

This next slide will show the testing curriculum path. As you can see here, the path starts with T101, Introduction to IST Standards Testing. And this provided an overview of testing in general. It could be applied to anything in this case but, in particular, ITS Standards. T201, How to Write a Test Plan. Cover the methodology to write a test plan for field devices and how this fits into the project lifecycle process. Now, the third module, T202 Overview of Test Design Specifications, Test Cases and Test Procedures explained how the three test documents are used to add details to the testing overview provided in the test plan. And the last three modules could be taken in any order. The first one listed there is T311, Applying Your Test Plan to the NTCIP 1203, version 3, Dynamic Message Sign standard. And then we have the subject of this webinar T313 Applying Your Test Plan to 1204. And next would be T3xx is listed here Applying Your Test Plan to NTCIP or TMDD or ATC standards. This would be some webinars which have not yet been created but which will follow along in this same testing curriculum path.

Now, the learning objectives for this particular webinar describe what you’ll be able to do at the conclusion of this course. You’d be able to describe within the context of the testing lifecycle the role of test plans and the testing to be undertaken. Next, you’ll be able to identify the key elements of NTCIP 1204 that can be used in creating the test plan and the other test documents that follow from the test plan. Next, number three would be the ability to describe the application of a good test plan to an environmental sensor station system that’s being procured. And finally, actually be able to describe a process of adapting a test plan based on selected user needs and requirements.

So first let’s start with a quick activity. In this case, why would an agency need to remotely monitor environmental conditions? So let’s start with where would the agency or what requirements would an agency have for needing to remotely monitor something that’s happening far away from where they’re located and what those environmental conditions are. So if you want to go ahead and start typing into the chat pod for any of your answers. Okay. We’ve gotten a response, assist with traffic management. So in that case we’re looking at actually providing input into the traffic management system based on environmental conditions. That would be an excellent one. To help in winter road maintenance is another. This obviously applies to some states more than others, Florida, probably not so much. Flooding conditions to divert traffic in the event of something happening there on the roadway you may want to inform people, motorists, in particular, of a hazardous condition ahead. Okay. These are all excellent reasons for setting up a remote monitoring system.

So let’s go through the answers given here. And one of them is to extend monitoring to locations that are not covered by weather bureaus. In this instance what we’re looking at is it may be that there’s all ready an existing monitoring system available to the DOT or to the agency which could be used to provide some of this information. So it may that the information that’s currently available does not cover all of the locations that are needed. And so the DOT would simply be looking to extend that monitoring to these remote locations. I can think, in particular, of some of the roadways that are going through mountain passes or far away from city areas where a lot of the weather monitoring would be easily found. Another one that was brought up would be inform motorist of hazardous driving conditions. So the particular example that was listed was flooding conditions but it could be anything from a possible bridge out to bridge flooded I should say, which could be monitored by sensors. A mountain pass that could be blocked. Or a fog or mist, anything that could be hazardous to motorists. Another example that’s just been brought up would be high winds. There are particular locations where high winds are known. I can think of a pass in Oregon where it’s known that these conditions exist but they don’t occur constantly. So it’s important that motorists know when these conditions do exist and can take them into account. Next item would be to determine when to close a mountain pass, bridge or overpass due to winter conditions? Again, we’ve got a location where an intermittent condition could cause hazardous conditions. And in this case, they’re so bad that DOT may wish to close some means of transportation. And in particular, in this case, it would be due to winter conditions, snow, ice, et cetera, rather than have someone have to go out to that particular location. The remote monitoring would allow them to do it all remotely, hopefully, even actually close the pass, using a dynamic message sign. And finally, it goes back to the answer we saw to help in winter road maintenance is to determine when to deploy snow plows or salt and sand trucks can be done through monitoring of environmental sensor stations. All of the information that’s being transferred back to the central system, could also be gained from the trucks themselves. There are many systems where the operators actually enter information as to current road conditions. There’s temperature and road temperature that can be automatically retrieved and all of this can be sent back along with location to the central system which would then allow for up to the minute information on road conditions. These can then be used to determine where to actually send the snow plows and how much salt and sand to put down. Or whether to put salt or only sand, depending on the temperatures that are actually being seen in the field. So these are all excellent reasons for putting a remote monitoring environmental system out in the field. Note, that these are all examples of user needs.

So as to the environmental sensor station itself, there’s quite a bit that can be monitored and that is supported by the NTCIP standard as it is right now. Some of them obvious, the wind, speed, and direction, temperature, humidity and air pressure is what we refer to there or barometric pressure. Next, would be precipitation type and rate so we’d be monitoring snow, hail, sleet, et cetera. Many sensors have the ability to distinguish between these different types. The next would be snow accumulation. And this is referring to what are the actual effects of that snow as it has fallen. It may be that it’s striking the roadway or the side of the road and is simply dissipating melting right there as it hits.

There are NTCIP supports the ability to distinguish between snow accumulating on the side of the road versus snow on the roadway itself. So that would be a very nice information to know particularly if you are applying any kind of treatment product to the roadways to be able to determine how well that’s working. Next would be visibility, so is there any kind of mist, fog, smoke per chance that’s reducing visibility, in particular, in an area which is known for having that. There are some locations, roadways that are near factories, which have a propensity for causing fog in the right conditions, and really reducing visibility, causing hazardous driving conditions. Next would be a pavement conditions and treatment. I’ve all ready referred to that to some degree. I know there are several systems that have been deployed in the field for actually monitoring that everything from embedded system in the roadway itself to infrared laser systems which monitor remotely from the actual roadway. By remotely, I mean several tens of feet, not miles away. The next item would be radiation. In this case, we’re talking about solar radiation, how much sun is actually coming through the clouds, et cetera. And the water level could be used for determining if a roadway has been flooded or a particular location has been flooded. So it goes back to our ability to warn motorists in advance of a hazardous condition. The NTCIP standard here we’re referring to also supports a snapshot camera. In this case, it’s a bit different from the NTCIP standard for pan, tilt, zoom cameras which give you a full functionality and a lot of features. In this case, it’s simply the ability to take a single picture and send it back which is nice if you are monitoring such things as water level or a particularly hazardous location to be able to see what the conditions are. Sometimes a picture is worth a thousand words. And the final item listed here would be air quality. So in this case we’re referring to the ability to monitor particulate matter, or any kind of pollutants that may be in the air. So this was felt to be the best location for that kind of information, in particular, can be used for putting up notices on dynamic message signs to perhaps have people fuel over night-- at night, rather. Or actually curtail trips if they can to reduce the amount of smog that’s occurring.

So as far as the environmental sensor station itself, it goes by several other names. You may here it called a weather monitoring station. Another abbr is RWIS, R-W-I-S, which stands for roadway weather information systems. So all of these referring to an environmental sensor station. So why don’t we take a look at what the agency would go through in actually putting together a project, once they’ve determined that they do want to install an ESS. We would start with, in this case, what’s termed the V-diagram. This gives a great overview of the project lifecycle. Starting on the left hand side and working through to the right hand side, we can see at the very beginning we start with regional architectures. What’s all ready in place that can be brought into the system? And how would this system fit into that existing architecture. Next, we’d be working in a feasibility study and/or concept exploration just to see what is the best way of actually implementing the user needs that were originally determined? Next, a concept of operations would allow you to develop exactly how the system fits together, what would be involved. You can start looking from this point at are we going a bit far with this? Is there too much? Is there an easier way to perhaps gain the same functionality? And finally, work into the system requirements, actually, what is required to perform those needs that were listed. You can see this all follows down the left hand side of the V through what’s called decomposition and definition and we’re really breaking down those user needs into actual requirements and working from there.

Typically, the high level design and detail design shifts over into the manufacturing side having taken those system requirements that have been generated by the agency. And at the very bottom we can see the software and hardware development and field installation occurs. Working back up the right side what’s listed as integration and re-composition. Now, we can see where the testing actually occurs. Starting with the lowest level of testing, the unit or device testing. This would be the actual environmental sensor station as it is tested. Working up from there, we would have the subsystem verification. At this point we start adding in the central system. So it would be everything involved in the environmental sensor station in the field, plus, the central system that’s actually monitoring that environmental sensor station. What would not be included would be if there were dynamic message signs that were intended to be part of the total system. They would actually get their own subsystem verification. And if the intent was, for example, to inform motorists of hazardous conditions, that would actually be tested at the system verification and deployment stage where the multiple subsystems are brought together. In this case, it may be that flooding is simulated at an environmental sensor station. This goes back to the central which then determines the message to be displayed on the dynamic message sign. So that would be an example of the whole system working together. And the final stage of the testing would be system validation, which means you go back to the user needs and verify that those are being met. Following on from this, the system would enter the operations and maintenance phase.

There might be some changes in upgrades based on feedback or changes that occur in the field. And finally, retirement or placement of the system. Looking in the center of the V, we can see that there are several plans, test plans that are linking phases in the decomposition and definition side over to the integration and re-composition side, so you can see the detailed design is linked to unit device testing by the unit device test plan. Working up we see the subsystem verification plan, the system verification plan and finally the system validation plan there at the very top. So as you can see, there’s quite a bit of formal testing that is involved in this project lifecycle. So why perform formal testing? There are a lot of other ways of testing the system. It could be a demonstration, for example. Simply put together the requirements and then tell the manufacturer to hand you the system and show that it works. Well, in a demonstration, they would simply go through several steps and say, there you go, it works. But in formal testing there’s a full test procedure and a lot more work that goes.

So let me just ask what answers would you have for the question, why would an agency go to the trouble of putting together a formal testing and, in particular, formal NTCIP testing? If you want to go ahead and enter your answers in the chat pod. Okay. Well, one possibility. I’ve gotten several responses here and one is to ensure NTCIP compliance. Another is to make sure that all of the requirements are met and another is to verify the system is in compliance. So we seem to have all of those covered here. One is going up to the very top of the diagram. It would be to validate the system against the user needs. This can be done in a formal testing manner. Another is to verify compliance with the procurement specifications. And here, notice that we’re using the term compliance to show that the device in this case actually meets the specifications that are put out by the agency and this is in distinction to the last reason for performing formal testing which is to verify conformance to the standard. In this case, we use the term conformance in reference to a standard, the standard we’re referring to here would be the NTCIP 1204 standard. So there’s a slight distinction there between the compliance and conformance. That’s mostly just in terminology, but I want to make sure that we keep that distinction. So what constitutes the documents that are needed for formal testing?

Well, let’s start with the test plans. In this case, a test plan is a management level document that defines several things, answers several questions starting with what item is to be tested and when is it to be tested? How is the item to be tested? Who is to perform the testing? And in what detail is the item to be tested? So management level means the test plans are primarily intended as planning documents to provide enough detail to ensure that all parties involved in the testing are on the same page. The real details of the testing are provided in later test documents. As to what item is to be tested and when is it to be tested, one way to look at this is to divide the testing in to the individual tests that are going to occur during that project lifecycle. As far as the what item that needs to be tested, in this case, it’s obviously the environmental sensor station. That one is an easy one to answer. So back to the when. In this case, the unit device test covers the item and its interfaces. As I mentioned, the subsystem verification, test the item, any communications path that’s involved might be dialup or a wireless system perhaps due to remote location. And any other items that communicate with the test item, so that would be the central system typically.

Next, would be the system verification and that ensures that the entire system meets the system requirements. So as mentioned earlier, the example of a dynamic message sign system that’s linked in with a ESS subsystem that would form the entire system. And finally system validation is used to the system is implemented meets the original user needs. So typically system validation is separate from the verification testing and may take the form of motorist polls or analysis of traffic flow or agency evaluation. For how the item is to be tested, what we’re looking at here is NTCIP conformance testing. This typically takes the form of interface testing using NTCIP test software. Now, one thing to recognize in this form of testing is that a lot of it will require specialized equipment to simulate environmental conditions. As part of the test plan, you want to take that into account so that you have a good idea of everything that’s going to be involved in actually putting this testing together.

Next, who is to actually perform the testing itself? There are several possibilities. We could perform the testing using agency personnel. It might be an out of house expert or even a manufacturer’s representative. So notice that each of these have pros and cons. Starting with the agency personnel they’re definitely familiar with the user needs being that they are the ones who are defining this system. But they may not be as familiar with the technical details of the devices that are available. They may not take full advantage of what’s actually available or perhaps maybe pushing them to their limits as it were. An out of house expert definitely would be technically qualified, maybe expensive or unavailable.

Next, would be a manufacturer’s representative. They would definitely be familiar with the devices that they manufacture in particular, but they may not be as familiar with the user needs. So it may be that the agency would not end up with a system that is particularly tailored to the user needs that they have. In many cases, the best option is a combination of all three of these where the agency would bring in an out of house expert to actually perhaps define the testing. The manufacturer’s representative would be performing the testing or vice versa.

Next, in what detail is the item to be tested? One way to look at this is to divide the testing into possible categories. So we’ll start with communications. So does the unit being tested conform to the communications standard? And next would be functionality, does the unit exhibit the functionality defined in the specifications. Another category is the performance of the unit itself. How fast is it? Is it reliable enough? This would be something that would have to be defined, obviously, as to reliability. Capacity. In particular, these days, think of capacity in terms of the memory, how many records or how many files can the unit hold, how much information can it hold before it has to return that information to the central, those kinds of items. Next would be the hardware, what is the unit actually made from? The materials that form that unit? And what is their strength and resistance to vibration. So a hardware requirement example comes from NTCIP 2101. This is the standard that defines serial communications using RS-232. So you wouldn’t really expect to find a hardware requirement here but it does define the physical interface that is expected to be found on the device, defines that as a female 25-pin connector. So as to performance, a performance requirement, an example from the ESS itself, the agency can specify the maximum response time for all communication requests. And finally, the environmental which might cover the temperature, humidity, water intrusion, ice buildup, corrosive environment, but notice that all of these are items that the device must resist or be impervious to. So we’re not actually looking at measuring environmental conditions in this case, more like resisting them.

So now the NTCIP standards typically cover the communications and functionality. As I mentioned there are some performance and hardware typically not in the environmental. The majority of what we see in NTCIP is communications and functionality. So in review of the test plans, the test plan is a management level document that provides an overview of a testing that is to take place. Now, let’s see how the test plan links in with other documents. So from here, once a test plan is created, you can see we go to a test design specification. So this document is going to specify details of the test approach for either a feature or a combination of features. And it’s going to identify the associated tests that are going to be performed to ensure conformance for those features. Now, following from the test design specifications, the test case specifications are generated. This is a document that specifies the inputs, predicted results and set of execution conditions for a test item. You’ll see that after this is created, the test procedure specifications, these simply specify a sequence of actions for the execution of a test.

You’ll note that the test case specifications and test procedures specifications work very closely together. And we will see that in just a moment. As to which documents are actually included in the ESS standards and any of the what’s referred to as the data dictionary standards of NTCIP you see right off on this slide that the test plan is not included. Nor are the test design specifications included. These actually have to be generated by the agency. And one of the reasons for that is that based on those questions that we were answering, you could see that there’s a lot of details that are very specific to each agency. It may be that they have standard test plans and test design specifications that can be used for many different devices in particular the test plans, as to who they feel qualified to perform the testing, when that testing is to take place, et cetera. The test cases, however, you’ll note that they are included in the ESS standard. And they’re combined in that standard with test procedures. This is slightly different than I showed you on the previous slide where the test cases and test procedures are separate. That comes from IEEE-829. In the NTCIP standards they’ve combined those test cases and test procedures into a single combined document and we’ll see how that works as we work into some examples. So now we’ll actually go over later how to create each of these test documents.

But first, some more detail on how the test documents are related. So you can see here this is a test plan layout starting at the top. We have the test plan. And from each test plan there is a test design specification which is going to give more details of how the testing is to be performed. And from the test design specification you’ll notice there’s typically multiple test cases that are going to be developed. In this case, there are three test cases shown. And each of those test cases may be used with one or more test procedures. And the same thing is true for test procedures. There may be multiple test cases that are referencing those test procedures. You can see on the left hand side, test case one and test case two both use the same steps that are called out in test procedure one. Whereas, in test case three, it actually calls out two separate test procedures, which is to say it actually uses different steps to perform the same test case. So an example of that might be if you have the same initial conditions or the same condition that you want to test for but you have a different set of steps that must be taken based on, for example, the particular device that you’re testing. It may be that a prop style wind sensor would be tested differently than an ultrasonic sensor as an example.

Okay. So let’s take a look at NTCIP itself. You’ll note that it’s a family of communications protocol standards. In particular, what it does is actually allows for the standardization of the flow of information between field devices and central communication systems. For the environmental sensor stations themselves, there’s what’s termed two data dictionaries, NTCIP 1201, which is a standard that’s used for a lot of devices and it’s what’s termed global objects. And what it does for the environmental sensor station, in particular, is it defines time and reports. Reports are also known as logging and basically allows the value of parameter to be recorded based on the change of another parameter. So typically it’s used for diagnostics in the event that something is happening out at the location, you’re not sure exactly what it is that’s happening. You can set up reports to log that information for you automatically based on what’s happening in those parameters. NTCIP 1204 the one we’re referring to here actually defines the parameters that are supported by the environmental sensor station. There are some other NTCIP standards that actually define how the device communicates back to the central, how it receives information from the central and how it responds to requests from the central. You can see that there are multiple ways that the communications network could be set up. Obviously, the oldest system would be dialup still being used in many instances for extremely remote locations. Another would be a serial interface. Perhaps there’s only a local interface available but very common, these days is Ethernet. It may be across the Internet. It may be on a proprietary network that the DOT has set up either fiber optic. Or it may even be over a wireless networks not listed here. It may be that that actually looks like an Ethernet network from the perspective of the central or the device. So as to what the environmental sensor station standards provide for us, the first thing is DIALOG/S. It actually defines how the central interacts with the field devices, providing us with a template for how these parameters are passed back and forth. So it’s not just the parameters themselves. And what this does, in particular, defining the DIALOG/S and the objects ensures interoperability of the elements of the final system. Finally, the heart of what actually gets passed back and forth is what’s termed objects. This is the term that’s used by NTCIP for the parameters that are transferred. And, in particular, how it’s transferring these is using what’s termed SNMP, simple network management protocol.

It’s through SNMP that these objects are defined and basically all that an object does is it stores values. And what the central can do with those objects is one of two things. There are a couple of others but these are the primary ones. The central can retrieve information from the device, which is to say perform a GET on an object. Or it can store information in the device, which is to say it can SET an object. And here we have an example below, windSensorLocation.1. This is a read-write object which is defined in NTCIP 1204. And that means that the central can both retrieve that information and it can set that information. So you can see that the location of that wind sensor it might be that it’s saying this is located I-47 east bound at mile marker 17, something of that nature. And that would be information that you’d want to be able to set by the central so that it can be later be retrieved by any central that might want to look at that information. The next object that we have here is windSensorSpotSpeed.1. And you’ll notice that this is a read-only object. So the central rather will only be able to retrieve information from that and this makes sense. It really doesn’t make sense for the central to be able to tell the environmental sensor station how fast the wind is moving. That would be called a fan. So in this case, that’s a read-only object. So note the DIALOG/S as I mentioned here, they’re only included in the newer versions of the standards. They were created using the systems engineering process. So let’s take a look at the history of NTCIP 1204 and see why these elements were added. So starting here, these are the published versions. There have been three. On the previous versions starting back in 1998, version 1, this was created without the systems engineering process. So you’ll note that there were no user needs, no requirements, and there were no DIALOG/S that were included. Conformance for this particular standard was defined using what’s termed conformance groups. Basically, just took several objects, put them together and said these all should be-- must be supported by device if the manufacturer is to state that they conform to the standard for this group.

Moving forward into 1204 version 2 this was created with systems engineering process. So now you can see we have user needs, requirements and DIALOG/S included in the standard. And rather than using conformance groups the standard now uses a protocol requirements list. We’ll be going through that here shortly as to how that works, but a great improvement allowed for a lot more flexibility in what the agency is allowed to select. And there are also some additional objects that were added to supersede some version one objects. And finally, the current version, version three, still obviously included the user needs, requirements, and DIALOG/S. But in addition, the major item that was added was the test cases and test procedures. And there’s a traceability matrix to show which ones are actually required to be used. And so the conformance, in this case, is defined through these standardized test procedures as they are called out from the protocol requirements list. So why is version three recommended?

Well, because the test procedures are published, they facilitate testing to ensure conformance to the standard. So the major benefit of this is that device manufacturers, system integrators and users can all independently verify conformance to the standard. Now, this is a really critical step because it allows everyone to do this and at their own timeframe, in particular, once a manufacturer has put together an environmental sensor station, tested it, deployed it, and is going to deploy it again at a different location or in addition. That testing will have been performed and it can then drop into another system with the minimal amount of problems. In particular, what we’re really looking for is reducing the possibility of incompatibility in the final built system. So that’s why we recommend the use of NTCIP 1204 version 3 there with the test procedures.

Okay. As to what NTCIP 1204 provides for us in creating these test procedures, there are several elements that are useful starting with the protocol requirements list. I mentioned this earlier. What it basically does is maps the user needs to requirements and specifies which requirements are mandatory or optional. So this was referring to that flexibility. This allows the agency to select the optional requirements and to know which ones are mandatory and which ones they’ll get automatically just by stating that the item must support NTCIP 1204 version 3. The next element that’s useful for testing is the requirements traceability matrix. What this does is associate each of the requirements with this dialogue and the objects that are used in that DIALOG/S. So next the DIALOG/S we went over that here just briefly previously. But this defines the standardized interaction between the management station and the field device. So not only is it important that the device and the central support the objects, and support them in the same manner, but also the interaction between the central and the field device occurs in a standardized fashion. It doesn’t show up so much in environmental sensor stations, relatively straight forward. It comes into play more often in some of the more advanced devices, a little bit more complex devices such as dynamic message signs or particularly traffic controllers. Now, the next element would be requirements to test cases traceability matrix. This actually provides a link between the requirements that have been selected through the PRL and the test cases that are defined in the standard. So this tells the agency and therefore the manufacturer and any integrators that might be performing testing which of these test cases and test procedures must actually be performed to show conformance. And finally, obviously, test cases and test procedures themselves.

So we’ll use all of these elements of NTCIP 1204 shortly in a case study, but first let’s look at an example dialogue. So as I mentioned NTCIP 1204 relatively straight forward. The majority of the requirements simply use the GET operation to retrieve information, what is the wind speed, what is your current temperature, et cetera. So in this case, it’s chosen a DIALOG/S from NTCIP 1204 that’s a little more complex, not by much. In this case, as I mentioned, the requirement defines what objects and in what order the actions that are being performed. So this particular one if you’ll notice looking at step B says the management station shall get the following objects for the pavement sensor of interest. And what we’re doing here is retrieving the pavement surface condition. So we’re going to retrieve three objects, the ESSsurfaceStatus. And this is going to say such things that might be in error. It’s obviously important to know but it might be that it’s dry or wet or shows possible ice, etc.

The next object would be essSurfaceTemperature.x. So that would be returning the temperature at the surface of the roadway itself which will be obviously most effected or would most effect the ability of the roadway to ice over. Finally, the essPavementSensorError. This would be an indication of if there is an error indicated in surface status, what is the particular error that’s being seen so we can get back some information. You know, if there’s an X at the end of all of those objects, this actually allows for multiple instances of those objects to be supported. So one environmental sensor station may be monitoring the pavement parameters at multiple locations, for example, multiple lanes across a roadway. There may be a location with multiple approaches to a bridge or multiple levels of an overpass system for interstate. Each of those would be subject to different conditions and it may be that one is going to freeze earlier than the others. You’d want to be able to monitor all of those. That X at the end is normally replaced with a number and that would be the actual instance of that sensor. So if you had, for example, eight sensors, then you would have essSurfaceStatus.1 through essSurfaceStaus.8. Okay. So let’s ask a question, now, here. So which of the following are included in NTCIP 1204 version 2? So remember there were three versions that were created. In this case, we’re asking what was included in version two? So you see we have several options. A, protocol requirements list. B, the requirements to test cases traceability matrix. C, test cases. D, test procedures. Or E, all of the above. Okay. And close the polls. And you can see here, that everyone has said the protocol requirements list is what’s included in NTCIP 1204 version 2 which is correct. You can see that the test cases and test procedures weren’t added until version three. And there’s not much of a point in having a requirements to test cases traceability matrix if you don’t have test cases. Okay. Next, let’s move into a case study. So this is going to show how to create a test plan and test specifications for a sample ESS implementation. And this case, what we have here is, well, a seemingly very remote location. And this particular instance, you can see the wind, speed, and direction is monitored by a weather vane and propeller style system at the very top. There’s a traffic monitor but that’s optional and a camera also optional. And then down towards the very bottom near the telephone pole you can see a temperature and relative humidity sensor. So what we’ll be looking at is just the wind speed and direction and temperature and relative humidity sensor for this.

So we’ll start with how does the specifying authority define the testing to be performed? And you start with the creation of a test plan. So what we’ll be doing for this example ESS installation is to start by creating a test plan to give an overview of the testing. And then we’ll use the protocol requirements list of NTCIP 1204 to create the test design specification. And, in particular, what we’ll be showing for that is the requirement to monitor wind speed. After this from the standard we’ll take the requirements, the test cases traceability matrix and will determine which test cases and test procedures are required based on the requirement to monitor wind speed. And then finally, we’ll define the inputs in the test conditions for individual test cases and test procedures. So if you want to follow along, you can go to page seven of the student supplement and there we have a test plan built out for this particular environmental sensor station. And what we’ll be doing is staring with creating the test plan. So remember the test plans cover the what, how, who, when, and what detail of the testing.

The individual sections that answer these questions and make up the test plan are defined in IEEE-829 and we’ll be going through those here. For this example, the test items are rather clear. They constitute the sensors and the controller that make up the environmental sensor station. The features to be tested are listed next. And note that in a test plan only the method of selecting the requirements is defined. The details are actually provided in the test design specification which will follow. And by limiting this testing to only the required NTCIP features this now constitutes a NTCIP conformance test. The features not to be tested which are listed next would constitute a compliance test which is performed separately from this test. Finally, the approach is defined which details how required test cases are selected based on the PRL and the requirements the test cases traceability matrix. Some of the next things to define would be the item pass/fail criteria. In this case, it’s for the entire item, not for individual features of that item. Those are defined at a later time.

Next, would be suspension criteria and resumption requirements. So in the event that the testing has to take place over multiple days or it may be that there’s a break during lunch or a break when something, in particular, has to happen, some events take a while to occur. Like, for example, monitoring rain over a 24 hour period, those should be defined. Or it may be that in the event of failure how can you simply restart that test? Or is it required to run through the entirety of the testing, how that retesting is to occurr should also be defined. Now, the test deliverables are defined at this stage. And this should include the format of the documents that are being provided to ensure that they can be read at a later time. In some cases the NTCIP testing is performed by test software. Is it such that is the output of that software such that it can only be read by that test software? Or does it have the ability to generate PDF documents or Word documents or something of that form that would be easily ready by other test software. Next, would be the testing tasks being defined and this promotes coordination between all of the parties. And finally, the environmental needs.

So this is going to define the physical space, tools and communications environment that’s needed. So now, we’re actually getting into the details of what’s involved in the environmental sensor station. So here we’re going to actually have to define what’s actually required to perform the testing itself. And, in particular, depending on where the testing is being performed you may need a large space, a smaller space. Typically, the testing is going to be performed using a data analyzer and test application. But there also may be some additional tools necessary. We’ll see that in a minute. So as to responsibilities this defines who’s responsible for performing these tasks. So the responsibilities that were defined in the test plan would be the agency project manager, the test analyst, the test manager and the developer. So all of these should be defined in the test plan. Staffing and training needs are defined at this time, define the qualifications of the staff required. So it may be that all of these can be done in house. These particular people would have to be defined. But if they’re going to be out of house, you’d still have to define exactly what is required of these people.

Next, the schedule is defined. And this really just defines the timeline in general terms which is to say we expect this testing to take two days, three days, et cetera, rather than specifying a specific date. It may be that it says must be performed before a certain date, but wouldn’t generally in the test plan stage, not give a specific date. Next, would be the risks and contingencies which should be looked at. It’s basically just describing what happens if things go wrong. So it may be that something is not available, either test equipment or personnel, something of that nature. Is there any backup plan for someone going on emergency leave of some sort? What effect would that have on the total project? And finally, approvals to verify that everyone is on board. Okay. The next document would be the test design specification and as you recall this document specifies the details of the test approach, in particular, for a feature or a combination of features. And it also identifies the associated tests that would be performed to show conformance for these features. So what the agency does to create the test design specification, it’s supported by NTCIP 1204 is to select the options that are necessary to meet the functional requirements using the protocol requirements list. Next, which test cases are necessary can be defined using the requirements to test cases traceability matrix. So in the next few slides, we’ll show how the PRL is used by the agency to select the functional requirements. So we’ll start with as you mentioned, we’re looking here at the bottom of this slide, at the actual features that are defined in the PRL. So you can see in this case we’re looking at retrieving ESS characteristics as a functional requirement. Looking at the column labeled conformance, you can see there’s an M listed there, which means this a mandatory requirement. And so the project requirement column indicates yes.

So this is a requirement that every environmental sensor station that claims conformance to NTCIP 1204 must meet. So you look down lower and you’ll see retrieve atmospheric pressure height is listed as NA and that’s because we don’t have a pressure sensor on our system. So this is an example of a mandatory requirement. Now, we’ll look at some other requirements. So in this case, as I mentioned, we could see that pressure was not selected. And one of the reasons for that is that it is what’s termed a conditional in addition to mandatory and optional there are also conditional requirements. And in this case, you can see that at the very bottom the requirement retrieve atmospheric pressure height actually doesn’t say just M or O, it says pressure colon M. And this is what’s termed a conditional requirement. The predicate listed there, pressure, actually links back to the section of the PRL at the top that says 2.5.2.1.1 pressure in parenthesis. So in this case, you’ll see, that this project requirement was selected as no. Now, this is an optional requirement. It’s not required for conformance to NTCIP 1204. So the agency had no requirement for atmospheric pressure so they selected no. And because they selected no that means that conditional pressure is false. And it means that the requirement, retrieve atmospheric pressure height, which is conditional on pressure is allowed to be set to not applicable (NA). And the next possibility is that a requirement may be optional. So in this case, you’ll see that we’re looking at the actual ability of the unit to monitor weather conditions. So these are options. They’re listed as an O. But in addition, there’s a lot of other numbers and symbols and you name it after that O. And these are what’s termed conformance groups. So in this case there are several conformance groups. Each conformance group has its own number. So there’s conformance group 0.1 at the top. That one’s been selected. The next conformance group would be 0.2, monitor weather conditions. Yes, we’re going to monitor some weather conditions so we selected that yes. And now we’ll look in detail at monitor atmospheric pressure 0.3. And you’ll see there’s a parenthesis 1..*. This indicates that from that conformance group you must select at least one of those options, that’s the one at the beginning. Then the star at the end indicates you can select as many of these as you like. So in this case you’ll note that monitor atmospheric pressure is not selected. It says no. But if we go farther down in that conformance group to monitor winds, 0.3 then that one has been selected. So that meets that requirement of selecting at least one of the requirements from that optional conformance group 0.3. So you’ll note that because that option has been selected, monitor winds, there are now multiple requirements below that that are mandatory. So retrieve wind data, that’s now mandatory, and that’s been selected in the project requirement. And required number of wind sensors is also mandatory. And it has been selected with a yes. Over on the far right you’ll see under additional project requirements there are sometimes items there that are listed that allow the agency to further specify information about the system that they’re specifying. In this case, what they’ve said, is that there shall be at least one wind sensor. And that’s what we had at the very top of our tower, so not a problem there.

Next, and once the protocols requirements list has been filled out, we’re going to look at determining which test cases are required, using the requirements to test cases traceability matrix. So as you’ll remember, what this does is links the requirement one or more test cases that require to be performed to verify conformance to the standard. So, again, we’ve still got that same section of the protocol requirements list at the very top of the slide. And what this is linking is through the functional requirement ID. So you can see it on the left hand side of the slide there, the box in red, 3.5.2.3.2.2. An arrow and it’s going down to the requirements to test case traceability matrix. Now, these are in two separate sections of the document, but this functional requirement ID links the two. So now we can see that same functional requirement ID 3.5.2.3.2.2 is linked to retrieve wind data. And the test case that that links to is C.2.3.3.3 which nicely enough is also titled retrieve wind data. So in this case, it actually tells us that there is one test case that must be performed to meet that functional requirement, to show conformance for NTCIP for that functional requirement. It may be and typically is that there might be more than one test case that’s required to show conformance for a single functional requirement. For these they’re usually relatively basic and a single test case is perfectly fine. Okay. So this is an example test case that we have here retrieve wind data. So what we can see is in NTCIP actually takes test cases and tests procedures and links them together. At the very top, we can see that this is test case 3.3 with a title of retrieve wind data. And the description shows that this test case verifies that ESS allows a management station or a central system to determine current wind information. So there’s a variable list of their required wind sensors. And you’ll remember, we saw back in the protocol requirements list there was a requirement of at least one wind sensor, so that would be the variable that’s set there the fact that there’s one required wind sensor. And what this is doing, you’ll see that we’ve in the test procedure section actually cut out a bit, gone straight to step five, which is the heart of the matter as it were. And what it’s going to do is perform steps 5.1 through 5.22 for all of the required wind sensors, in this case, there’s just the one. And it’s going to get all of these objects in step 5.1 wind sensor average speed, average direction, spot speed, spot direction, gust speed, gust direction and situation. As you’ll recall the N at the end of those objects simply indicates which instance of that wind sensor we’re going to be retrieving the information for. In this case, we only have one so all of those would be changed to dot-1 in the actual software that’s retrieving that information.

Next, step 5.2 is going to verify the response value is greater than or equal to zero. The standard actually defines each of those objects and says a negative wind speed doesn’t make a lot of sense. So that would be a failure. Next, we’re going to verify the response value that should be less than or equal to 65535. That’s actually the highest range available for that particular object. So any value that’s returned higher than that would also be a failure. And finally, the final step here says verify the response value for wind sensor, average speed, dot-N (.N)is appropriate. Wow. That’s kind of broad, that statement there. How do you determine whether that response is actually appropriate? So let’s go to how is response value determined to be appropriate? We have here several possibilities. And let me go ahead and bring this up. We can start with A an estimate of wind by on site personnel. Or B by monitoring simulating the output of the sensor, for example, it could use a variable voltage to simulate the output. Answer C, by simulating an input into the sensor. For example, a motor used to rotate the propeller of the wind sensor if that’s the particular kind of wind sensor that’s being used. It happens to be the one that’s being used here. Next, D, you could control the environment that the wind sensor is located in. So you could use a calibrated wind tunnel. And next E all of the above. So any of these could be used.

Okay. So if you want to go ahead and click in your votes, for which of these you feel would allow you to determine whether that response value is appropriate. Okay. And going to close the voting here. And we have several folks who say that the answer is all of the above. And one who says by monitoring or simulating the output or several I guess maybe, simulating the output of the sensor. So yes, B is correct. But also it may be that A, B, C or D could be used. So the answer is E, all of the above. And you’ll note that what we’re looking at here is the kind of in order of which one gives you, I would say the most rigorous test. But they’re also in order of which ones are most expensive, you know, actually, placing the unit in a calibrated wind tunnel going to be quite expensive. Estimate of wind by onsite personnel not expensive, but you’re only going to get so good a result. So it may be that the answer is somewhere in between. Or it may be that there’s some alternate means of determining that. But that’s one of the requirements that’s been placed on the agency is to actually determine how that appropriate responses are determined.

Okay. So moving on from here. We’ll actually look at creating the test case specifications. Remember that a test case specification as defined by IEEE-829 is a document that specifies the inputs, the predicted results and a set of execution conditions for a test item. So note that only a single instance of NTCIP 1204 version 3 test case 3.3 is required to verify conformance to the standard. Whereas more instances may be required to verify compliance with the project specifications. So as to the execution conditions themselves, this may define things like temperature, wind speed or humidity. The steps themselves are defined in the test procedures and we’ll get to that here shortly. Well, let’s go back to the difference between IEEE-829 versus NTCIP. Here, we’re showing test case 3.3 retrieve wind data from 1204 version 3. Note, the steps are referenced as the test procedure working in strict conformance to NTCIP. Selecting NTCIP test cases by means of the requirements to test cases traceability matrix would result in each NTCIP test case only being run once. But it may be necessary to run each test case multiple times with different inputs or test conditions. So we’ll now show an example of how this can be accomplished. So for this example, the NTCIP test case/test procedure is run twice, once when simulating calm conditions and once when simulating hurricane force winds. So you can see the expected results are shown in the table for the two objects that would be retrieved from the device. So you can see what we’re running the exact same NTCIP test case, twice, the first time for the top row indicating simulated wind speed of zero kilometers an hour. And we can see the results that we should be getting the wind sensor spot speed should be zero and the wind sensor situation should be returned to the value of three which is interpreted as calm. If we then change that to a simulated wind speed of greater than 118 kilometers an hour then we should then get a wind sensor spot speed also greater than 118 kilometers an hour. And the wind sensor situation will return a value 11 which represents hurricane force winds. So in this case what we’re showing is there are two test cases. Now, they would use the same test procedure. But remember in NTCIP they’ve actually put those two together. So it’s a little more difficult to indicate directly that you’re running that test case twice.

Okay. So which of the following are not defined in test case specifications according to IEEE-829? And we have several possibilities here. So which of the following are not defined in test case specifications according to IEEE-829? A is inputs. B is the execution conditions. C are the steps to execute. D the expected results. Or E, all of the above. And note in particular that we’re referring to how IEEE-829 defines test case specifications. And what we show here is steps to execute is the answer and that is correct. The inputs execution conditions and expected results are all part of the test case specifications according to IEEE-829. And so the one item that’s not listed there steps to execute that’s actually what’s defined in the test procedures specifications which is what we will go to next. So the test procedure specifications these define the sequence of actions for execution of the test. One of the purposes of standard test procedures is that they ensure that the conformance testing is performed in the same manner on separate test occasions or when they are performed by different stakeholders such as manufacturers, integrators or users. So note that it’s important not to skip any steps in the test procedures to ensure proper conformance testing. So again here’s an example of the test procedure and per IEEE-829 definition but only to find the steps. So in this case we can see the bottom half of this NTCIP test case and test procedure would be the test procedures themselves which is to say the steps that are being performed. So in addition to the test plan, test design specifications, test case specifications, and test procedure specifications, there are several other additional test documents that can be used. And we can start with the test item transmittal. This can be used to actually define when a test item is being transferred between entities, in this case, for example, between the manufacturer, and the agency. Or perhaps between the manufacturer and a testing agency if one is being hired. And what this is going to do is it’s going to define what is being transferred. And it also includes the status of that item. This is important particularly when you get involved in re-testing or, in particular, it defines what is to be expected of the agency that’s receiving that item. So, for example, if an ESS is being sent to the test lab it’s important that they understand this item has passed all of these manufacture tests and is now ready for the formal testing and, in particular, what formal testing is being performed. Is it just the unit device test or is it a subsystem test, etc.

Another item would be the test logs. This is going to document the testing that occurred. In many instances this is actually an automatic function of the test software itself. In another instances, the test software may have additional test monitoring software which would perform this test logging function. And as I mentioned earlier it might be that it’s important that the format of these test logs is actually such that the agency can later actually use these and view them to verify that the testing occurred as they felt it did. Another test document will be a test incident report. This will provide a means of recording anomalies that occurred during the testing. So it may be that a power outage occurred halfway through the testing and you’d obviously want to indicate that. It could be more mundane and simply be indicating exactly the conditions that occurred when an item failed. This could be very useful to the manufacturer in the event that they weren’t able to actually view the testing so that they could then recreate those particular conditions and determine why the item failed.

And finally, the test summary now, this is typically a short report that provides the results of the testing. Hopefully, it’s simply the item was tested and it passed. That would be the nicest form of test summary. But it may be that it includes more information about exactly what occurred, how those results were obtained. But, again, we’re looking at a management level kind of item which the test summary should be able to provide that information at a very short amount of time. Okay. So now let’s take a look at what have we learned today? We now know how to apply a test plan to the ESS standard. And we’ll have a bit of a question-and-answer here, a fill in the blank as it were. The test plan defines that testing be performed from blank. In particular, what kind of a perspective, if anybody wants to type in answers here in the chat box. So the test plan defines the testing to be performed from what kind of a perspective? So here it is the management level perspective or a high level perspective. Also, learned that ESS test plans cover the what, how, who, when and to what detail of the testing.

Next, what test document details the testing to be performed? If you want to type in an answer in the chat box, as to what test document details the testing to be performed. That’s the test design specifications. And next, what text document defines the inputs, outputs, and test conditions that apply to testing and ESS feature. That would be the test case specifications. And next, what test document defines the steps to be performed to test in the ESS feature. And that’ll be the test procedure specifications.

Next, we’ve learned to identify and use key elements of the NTCIP 1204 version 3 standard to standardize testing. We’ll start with what element that’s in the NTCIP standard shows the requirements that have been selected for a project. You know, this also indicates whether a requirement is mandatory. And that would be the protocol requirements list or PRL. And next what item that’s in NTCIP 1204 defines the test cases that are necessary to verify conformance to the selected requirements? And that should be the requirements to test cases traceability matrix with the long abbr RTCTM. Then, we learned to link the PRL to the requirements to test cases traceability matrix. Okay. And where to find more information? There’s quite a few documents available here. We’ll start at the very top with the Systems Engineering Guidebook for Intelligent Transportation Systems version three. You’ll note this was where the V systems engineering model is found. So there’s a lot more information there about the project life cycle how it fits into the testing that’s performed. But also is a great overview of intelligent transportation systems and the systems engineering process.

Next, would be IEEE standard 829. And you’ll note this is the IEEE standard for test documentation. And you think wow, we’ve been testing devices. But as we saw it works very well for formal testing of just about any device in addition to software itself. Next, there will be several NTCIP standards. As we mentioned the NTCIP 1204 and 1201 those are the data dictionaries that are associated with these environmental sensor stations. Next, would be the NTCIP 8007 document. This is where the actual testing documentation is defined for NTCIP. And finally, the last, certainly not the least, NTCIP 9001, the NTCIP guide, highly recommend this. Gives a great overview of the NTCIP family of standards. In particular, it gives some really, really detailed information on the actual information that’s passed back and forth between the central systems and field devices. So it gives a really great overview of how that all fits together with some great examples.

So anyone have any questions you’d like to bring up, feel free to go ahead and type those into the chat window. It could be on environmental sensor stations, the testing of environmental sensor stations or the NTCIP standards in general. Anything that you’d like to have answered, feel free to go ahead and type those in. I have a question here. And starting with how would the process be different for purchasing off the shelf management and field stations versus custom development? Okay. What we’re looking at here is, well, a combination of an acquisition and a testing question, starting with the acquisition. In particular, you’re still going to do the systems engineering process the same, which is you’re going to start from a user need. As long as you have that clearly defined if you know exactly what you want the system to do when it’s put in place then you have the ability to actually look at off-the-shelf systems that are available and see if they perform the functions that you really need. Once that has been done and you have your user needs, you’re going to be working through the rest of the project life cycle as we saw it. But, in particular, you’re going to want to be very specific about the system requirements. And make sure that any off the shelf management software or field stations are going to meet the requirements that are necessary. Obviously, one of the benefits of an off-the-shelf solution is that it should be less expensive, hopefully, and it may be that it has all ready been tested, all ready installed other places, should drop into your system much, much easier. The disadvantage of that would be that it may not perform all of the functions that are required for what you’re doing. And by having to find your user needs early you’ll know what’s actually not included there. It may be that you’re willing to forgo some items. You know, another item off-the-shelf hopefully should be less expensive. So it may be that based on budget constraints you simply have to pass on that. As to the testing itself, we’re talking about NTCIP conformance testing, should be pretty much the same. It may be that if you truly are getting a system that’s identical they may all ready have NTCIP test results that they can simply present and show that item is fully conformant. If there were any modifications that had to be made, then it may be that you want to perform either sections of the test or the entire test over again at least for the first one that you would receive. But otherwise, it should be very similar. So is there any other questions that anyone has about ESS, ESS testing, NTCIP? Okay.

Next question would be how do you determine the level of testing appropriate given the cost and criticality of the system? I think that definitely kind of almost answered the question yourself. Obviously, the more detailed the testing that’s to be performed, the more expensive it’s going to be, the longer it’s going to take, et cetera. And it’s all going to be dependent on exactly what that system is intended to do. You know, it may be that it’s performing , I won’t say a noncritical but an element that would not be effecting motorist safety directly. So it may be that there is the ability to withstand a relatively low failure rate. An example I’m thinking of which is very different, where motorist safety is definitely involved would be a visibility system, where you’re monitoring for just very, very low visibility that comes out of no where. You can be driving on a perfectly clear day, come around a curve, go down into a valley and suddenly you’re in complete whiteout conditions. And so in that case you’re looking at a much, much higher risk of danger to the motoring public. And so you’re going to spend much more in terms of the testing of that to ensure that the failure rate is extremely low. And you also want to actually not quite as bad to have false positives as it were, if it indicates that there’s possible fog ahead and there’s nothing, that’s perfectly fine. But having it miss any kind of fog may be very, very hazardous. So all these things have to be taken into account when actually designing the testing for these systems. Luckily through the NTCIP with it having the ability to perform the testing in a standardized fashion in many cases, manufacturers are more willing to go through this testing because they know that they can, perhaps, have it done by an independent agency to verify conformance. And then perhaps do it in a more thorough fashion through everything they’re doing. Have it done once and then have that to be acceptable to multiple agencies.

Okay. Any other questions? Another question, if you’re going to install ten ESS stations in one project do you test all ten of them with the full test or test one and hope the rest work? Ooh, good question. Typically, NTCIP testing is performed on a single unit. And in this case, what I’m referring to would be the full suite of testing, including failure modes, et cetera. But what you’re going to want to do that’s going to constitute your NTCIP conformance testing. Typically, what an agency will do after that is put in a testing of each individual unit as it’s received and installed in the field. But it’s simply going to test for, in essence, the hardware. So, for example, under the wind sensor and wind speed, there are multiple objects that can be retrieved that deal with the spot wind speed, how fast is the wind going right now versus the average wind speed, how fast has it been going over several minutes. Versus the wind gust, what was the fastest wind speed that occurred in, I think, the past five minutes. So you can get a feel from that as to what the wind quality is out in the field. But you’ll note that once the system has been qualified to be conformant to NTCIP and you know that if the system is retrieving valid wind speeds that it would then have the ability to make all of these individual measurements and, in particular, measurements that occur over time. So it may be that there is a smaller subset of test cases that are performed once the unit is actually placed in the field to verify that it’s been properly installed because that’s really what you’re looking for at that stage is that each of those ten units would be installed properly and don’t differ in some fashion from the one that was actually tested during the NTCIP conformance testing. But yeah, it’s a great question. So anything else? Okay. Then this ends the ITS Standards Training Webinar T313 Applying Your Test Plan to the NTCIP 1204 version 3 ESS Standard. Thank you all for attending.