Transcript of Question and Answer discussion (post-presentation)
Mac Lister and Emilano Lopez answer questions compiled during the presentation submitted by attendees. Mac Lister reads the questions.
Q: Mac Lister: Why is field installation at the bottom of the V done before many of the testing on the right side of the V? You want to take a first crack at that one, Emiliano?
A: Emiliano Lopez: Yeah. We even talked about this a little bit as far as it all depends upon your agency. A lot of folks are okay with doing their testing even to the left side, where they'll do factory testing. But for a majority of the agencies, when we talk about field deployment, they want to check the system in the field, operational. So in essence what you're getting is your getting a pretesting in the factory to make sure that when you do your unit testing and even at that point, it may be in a controlled environment, an integrated system. For a lot of agencies, they want to see the system in the field and that's when they want to start seeing testing done, to make sure that it is going to function as expected under the field conditions with the communications of the structure and with the hardware that it may again be constrained by.
Q: Mac Lister: We have a question about that there is no document approval that's shown between the feasibility study and concept exploration and the concept of operations in the V diagram.
A: Mac Lister: I had talked about the small purple kind of circles as gateways for checkpoints in the V diagram and there is none showing between the concept exploration and the concept of operations. I guess my answer to that would be that the feasibility study, concept and exploration are on the wing of the V. That happens prior to the project actually being initiated. The first step in the project and the feasibility study concept and exploration have more to do with justifying and scoping the project. But once the project actually begins in the first step of the V concept of operations, that's when these documentation gateways occur. There would certainly be documentation that's produced out of a feasibility study or concept exploration. If you looked in Section 4.2 of the handbook, it does talk about that there's a review and approval of the feasibility study from project management that occurs. But what we're calling the gateways in these little purple kind of elongated circles are more within the project itself once it's been initiated. That would be my answer. Emiliano, do you want to add to that?
Emiliano Lopez: Yeah. I guess that concept is taken from again, the once through waterfall or the linear approach to doing a project. One of the things to recognize in system engineering in the V is that it's iterative. There's a lot of back and forth going on. In fact, you may get through the entire V and as you come up to system validation, it may be completely different from your concept of operations initially. So what you end up doing is actually going back to your ConOps and changing that. The reason why there isn't a bar between those two points is that at the feasibility study concept exploration stage, you're just coming up with again, a generic way of doing this. As you go further into the concept of operation, it's feeding off of that. In all reality, those two are sometimes done at the same time.
Q: Mac Lister: Okay, the next question, apparently from a frustrated project manager, is: How does an organization keep up with ongoing technological changes?
A: Mac Lister: My first off the cuff answer is welcome to the 21st Century. That is, indeed, challenging. I guess in a more serious vein in answer to your question, is the resources that are available through this handbook will continue to be upgraded. The lessons learned database, the other kinds of resources that are listed. You can go through our peer-to-peer program through the division office to find out what some other states are doing. You can talk to people like myself and the Resource Center whose job is to keep abreast as much as possible. And you can talk to your consultants and contractors. They certainly are aware of the technological changes as they impact the systems that they're designing and developing. They're a good source of information as well. So beyond that, I don't have any magical way of doing that. It is a product of the 21st Century and rapid changing technology. Good luck. Emiliano, do you have any words of wisdom you want to add?
Emiliano Lopez: Yeah. I think that is the whole crux of why system engineering is needed. Don't focus on the technology; focus on the needs. When you do that, there are a lot of system engineering processes and tools that will help you negate the fact that technology is changing, everything from the alternative analysis, which evaluates existing equipment to configuration. System engineering and our philosophy is don't focus on the equipment. As you drill down to that level of detail when you are ready to select technology, where there is a specific hardware or a specific technology. At that point, your analysis is going to tell you, based again on what your needs are, what your resources are to support that system, what you should select. Even within our guidance and suggestion of procurement, we talk about that, making sure that you have a contracting vehicle that doesn't limit you to a preconceived idea, a preconceived system, a preconceived technology or equipment vendor.
Q: Mac Lister: Are there any completed systems engineering analysis by other users that could be provided as a guide for three levels of projects such as a very complex project, a medium complex project and a simple ITS project?
A: Mac Lister: There are checklists that are available that are in the handbook itself. I guess the answer that I would give would be contact your division office. Because as I stated in my portion of the presentation talking about the systems engineering analysis, the actual documentation requirements, because of the statewide stewardship agreements between Federal Highways at the division office level and the individual state DOTs vary somewhat. The documentation requirements are specific to state. So I'm a little reluctant to give examples from one state that might not be applicable in another state where specific documentation requirement is required by that division office. So my answer would be contact the division office. Emiliano, do you want to add to that?
Emiliano Lopez: No. That's a good suggestion.
Q: Mac Lister: Okay. I'm going to give you the next one, because it fits your background, I think. Implementation and construction are two terms that need some clarification at this point in the discussion.
A: Emiliano Lopez: Unfortunately, we are working in the real world. So those terms are used synonymously. When we talk about implementation, of course we're talking about typically the system itself, the hardware and software or even development stage. When we talk about construction, we are talking about the actual in-the-field. Because that term has different meanings for whichever agency you're in, it has funding implications. For instance, when you start talking about construction, that defines what type of contracting vehicle that you're allowed to use even at the state level as well as with federal. I'm not sure that maybe the person who asked the question could actually or <inaudible> on why they need a better definition or what the issue is or confusion out in the field when using those two terms.
Mac Lister: I guess what I'll suggest is that person that asked that question could either type it in and we'll get to it later or contact you or me with a separate email. I'll be glad to explore that a little further.
Q: Mac Lister: Next question is operational scenarios in the requirement. Why are operational scenarios required?
A: Mac Lister: I guess I'll just state that in INCOSE and in other kinds of folks who have studied systems engineering methodology, they have found operational scenarios to be a really invaluable way of documenting how the system works at a local level. So although we don't specifically require that, it really is a good way to document how the system will work. Now generally, what we've said in our guidance is for those high level or really important portions of a system or portions of a system that are complex because of multi-agency involvement or that kind of thing, those might be the good targets for developing an operational scenario.
Emilio Lopez: Yeah, I'll just echo what you said. Operational scenarios aren't required. They're just one strategy, similar to storyboarding, when you're trying to come up with an idea. Operational scenarios just help people get a better feel for okay, what are we talking about here? That, a lot of times, is a starting point and can be built off of. A lot of the systems that we're talking about aren't going to be the be-all, end-all system. Given the timeframe that most systems need to be upgraded, there's nothing wrong with coming up with five or six scenarios for your need. Then as you identify additional needs through just use, it becomes almost a spiral model of getting the system built based on an initial premise. Then, as the system is used, you understand the additional functionality you need. Operational scenarios aren't meant to just come up with pie-in-the-sky things. They're really just a way to get a fundamental or a basic understanding amongst all the stakeholders on what we'd like the system to do initially.
Q: Mac Lister: How early in the SE process do you recommend engaging the development group?
A: Mac Lister: The answer is very early. The term "development group" is I'm not sure exactly what that means in the kind of methodology that the questioner has. If the reference is to that person or group of persons that is actually doing the detailed system design and building, which is the way I'm taking that question, they really do need to fully understand and participate on some level in the earlier tasks of concept of operations and requirements or at least have reference. If it's contractually a different group and you can't do that, at least have detailed reference to all of the documentation coming out of those processes. Emiliano, do you have a further comment?
Emiliano Lopez: No. That was very succinct.
Q: Mac Lister: Are all these documents required if the intent is to deploy an off-the-shelf system?
A: Mac Lister: No. What Emiliano said in the session talking about tailoring was don't skip steps, but know the step might be we've done that already or it's off the shelf. You may be adding devices to an already existing network and software package. If "off-the-shelf" means installing new software, the answer is probably yes. So it really depends, to some extent, on what is the deployment of this off-the-shelf system? How broad is it? How many areas and how many parts of the organization and how many legacy systems might it feed data to or get data from? There's not an easy answer to that. Emiliano?
Emiliano Lopez: I guess the analogy I always give is if you want a road built, it's pretty much off-the-shelf. Just build it and I'll be back in two years. The reality is you don't do that. Even though you know where the road's going to go and what the specifications are, in the contract you still need to have the quality assurance and quality control so that you can make sure you get your final product. That's the same with systems engineering and systems in general. Just because it's off-the-shelf doesn't mean they just show up and drop it on your desk. There is an awful lot of customization that goes on. They need to configure the software for your particular road network. There are certain levels of access. There are a number of things that need to be taken into account. Again, we're not saying that you have to do a ConOps and everything to the full extent. But what we are saying is for those things that you need to check, again system engineering is not anything new. It's probably the same things that you would have required for a roadway project or if you're buying new IT equipment, anything in that regards to where you have to have a common understanding between your service provider, your contractor-consultant and yourself. And that's what we're talking about.
Q: Mac Lister: How do institutional forces impact each major project step like design, not just technical considerations?
A: Mac Lister: I don't see them listed in the table or requirements. They're addressed in Chapters Five and Six more than in Chapter Four. To that extent, the handbook is stovepipe a little bit, but I think a number of the institutional issues that may be impacting a project are addressed in the applying systems engineering chapter or the individual project planning and monitoring and control steps that an organization has. Again, a lot of these questions can be taken to a lot of different levels. One of the things that I would say too, is for example, your concept of operations. If there are multiple agencies involved, you may certainly have institutional issues. Your regional architecture may have addressed that. Your regional architecture is supposed to have addressed what agreements need to be in place to implement the projects that are outlined in that architecture. So that may be another source of information. Or another forum where you can take the question of institutional issues and what agreements need to be drawn up back to.
Q: Mac Lister: Should the manufacturer provide the unit test procedures?
Mac Lister: I'll let you take a first crack at that one, Emiliano.
A: Emiliano Lopez: The answer is no. If they're providing you the-- well, I shouldn't say that. I should probably qualify that. If they're going to be responsible for developing that early on in the development process, so that it corresponds to your high level system requirements and as you get down into the detailed design document, the detailed requirements, yes. If they're planning on providing you that test at the end of the project, all you're going to be doing is getting a test plan of the system that they built and delivered.
Mac Lister: Right. And it also could be a project where they're delivering new devices to a network of devices that already exist. It really depends on a number of project level factors, as Emiliano discussed.
Q: Mac Lister: The confusion that we all have, I've seen this question in a number of places before, in the O&M acronym, the M has today been referred to as maintenance and management. Can you discuss the difference in these two terms? Are both correct in the systems engineering process?
A: Mac Lister: In the handbook, when we talk about O&M, which is the wing on the right side of the V, we are specifically referring to operations and maintenance. Management within the systems engineering framework is more the management processes in Chapter Five that relate to the individual project management. That's different than what's depicted in the systems engineering V. So management is also taken on a context in other kinds of discussions beyond project level where you're talking about management of the entire network or regional kind of construct that might exist. So it is confusing. What I would focus back on though for purposes of this handbook, O&M is operations and maintenance of that particular system, and management refers to the management processes involved in developing and implementing that specific project.
Q: Mac Lister: Examples of test plans.
A: Mac Lister: Are there some available? I'm trying to remember in that section, Emiliano, there are--
Emiliano Lopez: We do have some examples in our handbook, as well as California has some examples. However, I guess it's okay to look at them so it gives you an idea of what columns you want and what things that you want in your test plan. Maybe if that person could contact either one of us and tell us specifically what they're looking for.
Mac Lister: Yeah, it's Section 4.9 of the manual that addresses that. There are some available in that documentation as well as the California guidance Emiliano suggests. If that doesn't answer your question, feel free to contact us.
Q: Mac Lister: We are tasked by the state transportation agency to design and implement an ATMS subregional system that may involve ITS technology deployments on several roadway corridors. Due to the size of the project and funding restrictions, the agency determined that the subregional project needed to be broken into several smaller corridor projects that are phased over several years for letting. Assuming that the subregional ITS concepts and objectives remain unchanged, can a single system engineering document be prepared for all projects within the subregion, or does a separate system engineering document need to be prepared for each corridor?
A: Mac Lister: Well that gets into that tailoring question that we discussed and building on existing documentation. Without knowing the specifics and if there are specific issues that would need to be addressed as part of subregional areas, I can't give a definitive answer, but the general guidelines that we're presenting are don't reinvent the wheel. So to the extent that existing documentation can serve additional implementations, then that's a place to start that discussion from at any rate. Emiliano, do you have thoughts on that?
Emiliano Lopez: Yeah. A lot of it depends on who the project owner is. If, in fact, this is being all done by the state and it's all being done within one group within the state, it probably does make sense to have all that documentation in one document. It really depends on if that's the way it's configured that it's just one entity that's going to be controlling and monitoring and making changes. The configuration management we talked about, the best way to look at it is similar to the way some agencies do their traffic management system, their ATMS's. They may start out with just doing the traffic management center, but then it may be broken up into four or five phases, just different interstate corridors are expanded, more and more are put out. But in itself, the project documentation, the ConOps are all in one place so you can see how they all interrelate to each other. Now if they just happen to be all standalone systems like let's say it were a bunch of closed-loop systems in a given region and there's no relationship or interconnectivity there, you could do it either way. It might make sense to have it in separate documents for each one of those systems so that it doesn't pollute or it doesn't overburden the documentation of any one of those systems.
Mac Lister: Yeah, and the final answer then, of course, is you will need to do what the state agency requires. I'm sure that they'll make their decision based on a number of those factors that Emiliano has outlined.
(Q/A: Mac Lister: The difference between the systems engineering handbook and the California guidebook, I think we discussed that already. If somebody wants more detail, let us know.
Q: Mac Lister: For projects where the agency knows exactly what they want, a systems engineering process may seem onerous and expensive. Why, in this case, should the agency not forego federal funds and these systems engineering strings?
A: Mac Lister: Because only 33% of projects are successful. I really hope that I presented the benefits of good systems engineering and the need for good systems engineering as the best reason for doing systems engineering, as opposed to the federal rule. If the agency knows what it wants, that's a funny phrase. Does that mean somebody has written it down? Does that mean somebody has written it down in terms of how the system will operate and what's required of the system? They need to do that. They need to have a good concept of operations. They need to have good requirements. They then need to tie those requirements into testing to make sure that they get what they want. So I don't think of these as SE strings. I think of them as sound business practice. I can't say that any other way. Emiliano, do you have anything you want to add?
Emiliano Lopez: Yeah. I've been in the public sector since late 1980s. I'm not going to date myself. I've always been doing either signal systems or some sort of a traffic management system. In every case, we knew exactly what we wanted. But there are a lot of things that happened. Thought changes. Priorities change. And with all those changes, these systems engineering strings that you're talking about, they seem to be your only tie to keep in your face, in everybody's face what the intent was. I guess I just can't echo enough the importance of these system engineering strings that we're talking about here, how important and how valuable they are to your agency. If you can only look at them as something that's burdening you, look back at that one slide that Mac threw up there. That's not very encouraging. So it's all on how you look at it. If you're just looking for today and putting something in the ground, they are strings. But if you're looking at the overall cost and benefit to your agency, as well as to anyone else that may be coming into your agency, there's a lot of value to systems engineering.
Q: Mac Lister: Any example of risk management?
A: Mac Lister: Chapter 5.3 in the handbook addresses risk management. I just took a very quick look at it. It does not have any specific examples. As I think about it, the reason for that would probably be that determination of risk is very, very project dependent. It would really depend on the kind of project that you're talking about, the size, the scope, different integration possibilities that the system is exploring. If that questioner wants to send an email to either Emiliano or me with a little more description of the kind of project that they might want to find an example for, we might be able to come up with something. There may be also, I have not looked through the detail of all of the resources listed in Chapter Seven. There may be some examples in the INCOSE material or some of the other material that's listed that I would ask you to take a quick look at as well.
Q: Mac Lister: Is there documentation available that compares the tangible benefits, investment costs of systems engineering with cost efficiency savings gained in ITS projects by reducing cost add-ons and overruns?
A: Mac Lister: I'm not aware of any at this time. I'm hoping, as time goes on, that we're able to add those kinds of cost benefit information data sources to our lessons learned knowledge database, but I'm not aware of any at this point. Are you, Emiliano?
Emiliano Lopez: I guess this lies first, always say intrinsically system engineering proves itself, because we don't have, just because of the failure of all the ITS and advanced technology projects we have. One of the things I think, but at the same time, I understand what the state and local agencies are up against. To ask for additional funds to do systems engineering, you have to justify, maybe not with the final rule, but definitely you want to show the benefit of expending those funds to do the system engineering. Probably that's the best way to do it is show that cost benefit ratio of the system engineering to these cost overruns. We know they're there. The bottom line is when you do a project, it really isn't change orders, because in a lot of cases, it isn't just change order because I felt willy-nilly I wanted to do it. A lot of times it's change order because I didn't really invest the right amount of time and I really didn't think about this and I didn't really talk with everyone to understand ultimately what the system was that we wanted. So what I guess I'm saying here is you're going to pay for it anyway. But the point is did you want to pay the lower fee up front by identifying it before the contract was let, or do you want to pay the change order price? I think maybe that's something that can be done is showing those change order prices, by percentage, and show what that cost savings would have been to a project.
Q: Mac Lister: How could systems engineering be used to address discrepancies between project budgets and project cost estimates developed based on the concept of operations?
A: Mac Lister: There's a slide we use in the project management course that talks about the ability to estimate project costs at various points in the project development cycle with data that shows can be up to a 400% difference between what you estimate when you start the project and what you estimate after concept of operations, after requirements. In other words, the curve gets closer and closer to the actual costs as you increasingly define your system down the left side of the V. It's true that systems engineering does that. As you define your project, your project budge gets much closer to actual implementation costs, if you've developed a good concept of operations and developed good requirements. So I guess my answer to that question is that's what systems engineering is used for. That's how it's used to help control the costs of a project. Because at each step you define more fully what you anticipate the system to do and link it to the verification activities and the testing activities on the right side of the V. So my answer is that's what systems engineering does. Is there another aspect of that question, Emiliano, that triggers in your mind?
Emiliano Lopez: I guess what I'm reading there is in the normal STIP/TIP process, just raw numbers are thrown out there for a traffic management system. A lot of times they may be based on "Oh, Georgia or Virginia has this system and what did they pay for it? About $5 million. What do we want? Oh, cameras and dynamic message signs." So they come up with these just raw generic numbers that they want to get into the STIP or TIP, just so that they can get their foot into the funding cycle. But as you said earlier, again I'm just echoing what you said, Mac. Through the system engineering process, you can better identify that system that you ultimately want to build. So now instead of going with an off the rack number, you've got a customized number based on what you want to deploy, what functionality you need. The whole system process starts well before the development stage that we're talking about here. You can actually use it as you're getting prepared to submit projects into the STIP or TIP process. That'll take away that discrepancy that you talk about between the project budgeted at the STIP point in time versus as you start doing the actual project, the cost stats you come up with when you start specifying all the very fine details of the system itself.
Q: Mac Lister: When a system is designed for implementation in multiple agencies, is it best to have one requirements and design document or to have a separate document for each agency?
Mac Lister: You want to take a first crack at that, Emiliano? I have a thought I'm working on here.
A: Emiliano Lopez: It all really depends upon your institutional agreements and arrangements that again, we're down at the project level. So if there are cooperative agreements already in existence where it could even be what agency is going to operate the whole thing? You've got to pull back and take a look at resource wise, as well as KSA wise. Does it make sense to have it all under one agency who will be kind of the point person for the project, or to have it kind of distributed? Again, it kind of really depends on the relationship and agencies working on that particular project.
Mac Lister: The other thought that I have is are the multiple agencies operating the same way? Are the individual agencies verifying and paying for parts of the system separately? In other words, what I'm saying is the systems engineering V process all linked together? So if there's a different verification plan that different agencies might have or operates a little bit differently, then the same requirements and design documentation may not be appropriate.
Q: Mac Lister: Okay, next question has to do with design build. How far should the owner take the systems engineering process in conjunction with design build projects?
Mac Lister: I'll let you start with that one, Emiliano.
A: Emiliano Lopez: Design build is no different from doing the project yourself or doing it a hybrid of using a contractor-consultant. It's still your money. You still have expectations. What you're paying for in a design build is the fact that you have an unknown and you're leveraging your design builders to gain you some cost savings and time savings through innovation. Nonetheless, those same quality assurance and quality controls have to be maintained. Otherwise, you're not going to know what you get. Ultimately, your design build partner also has to have something written down as far as what the expectations are and how things are going to be verified. A lot of the same building blocks of system engineering. I guess the question really is to what degree are you doing stuff, versus the design builder? I guess the answer to that one is since you're paying for them to do that, you lay out the ground rules on how the system engineering's going to be performed, the deliverables and again, to your comfort level and they do all of that for you.
Mac Lister: The other part of that question that comes up for me when I read that is a concern that the agency stay involved to a significant degree. As Emiliano stated, there is some expectation for this system, even if you're basically turning it over to a design build contractor. And the agency needs to be involved at the various steps in the process to assure that the design build activities are occurring within the idea of how the system will operate and the generalized requirements that may have been presented to that design build contract. You can never turn it over totally to a contractor, unless you want the system that the contractor wants. If you're willing to say that, then you can. But none of us are willing to say that. Okay.
Q: Mac Lister: Do you have a sample state DOT contract that requires a private engineering firm to file all the necessary documentation for an FHWA funded project?
A: Mac Lister: That again, I think gets into the particular requirements of the FHWA division offices. So any example that we might be able to come up with would be dependent, to a great extent, on the requirements of the FHWA funded project as it relates to the state stewardship agreement with that individual FHWA division office. So if you want additional information on that, I would suggest contacting the FHWA division office in question.
Q: Mac Lister: Does the handbook cover recommendations on testing ITS technologies that produce subjective results, video produced by a CCTV camera?
A: Mac Lister: No is my answer. Emiliano?
Emiliano Lopez: Yeah, I echo that.
Mac Lister: The handbook really is a methodology for systems engineering, not specific evaluation of recommending on specific technologies. So it's technology neutral in that sense. But again, if you want more information on individual technologies, you can contact either Emiliano or me and if we can find some resources for you, we'd be happy to try and do so.
Q: Mac Lister: Why is there no backtracking arrow between the ConOps and the O&M boxes in the V diagram?
A: Mac Lister: When developing the ConOps, one better have an understanding of the O&M requirements. That's true. I guess the answer is that the backtracking boxes have to do, again this is the same answer that I gave in terms of concept and feasibility exploration. O&M is on the right wing of the V. So the arrows within the V process refer to the core development processes of the project. But that doesn't mean that you don't document O&M in your ConOps. You certainly do. So I think the questioner has absolutely the right idea. Emiliano, you want to add to that?
Emiliano Lopez: Yeah. The backtracking arrows are checkpoints. So in correlation to the ConOps, the backtracking arrow goes from system validation to ConOps. That's when you're done with it and you're at that activity system validation. During the ConOps, a validation plan is developed, as we noted early in Chapter Four. So once that validation plan has been developed, the system is put into operations and so that's where the testing is. Seeing if that validation plan does map up to what's really happening now that the system's up and running. That's what the backtracking of that arrow really means, back to the ConOps. Operations and maintenance is now putting it into everyday use. That doesn't negate the fact that as you began your process, operations and maintenance was taken into consideration. We mentioned that earlier, that the left and the right wing are really lifecycle processes into the design process itself.
Q: Mac Lister: Our client, a transportation agency, developed an ITS architecture a few years back to meet the FHWA requirements under Rule 940. All of the major stakeholders were involved in preparing the architecture document. Most of these stakeholders have capital programs that project out as far as a 20 year window. If the architecture was based on these long range capital plans, there would be minimum need to change the elements in the architecture, namely the project sequencing--short, medium, and long term plan. With that in mind, how often should the architecture document need to be updated, since the staff resources may be limited? Should we consider updating only the ones that provided updates, since those stakeholders have resources and not require others, ones with fewer resources to provide updates as often?
A: Mac Lister: As I interpret that question, it has to do with the need to maintain the regional architecture. In addition to the handbook on systems engineering, we have a guidebook on developing and maintaining regional architectures. I would refer the questioner to that document as well. That's also available through the same operations FHWA website. It does talk about maintenance in there. It talks about reasons for maintenance being planning activities occurring in the region, but also project implementation. So as projects are implemented, they may or may not address all of the issues that were in the regional architecture. Or they may address other things that the regional architecture didn't maintain. Our message is that maintenance needs to be looked at and considered in the regional architecture. In fact, it's a requirement of the regional architecture that the maintenance plan be specifically defined. So there should be in that region a methodology for project implementers to provide feedback to the regional architecture. Real honestly, we've found in reviewing regional architectures that some of them don't. And so our comment always is to the local area, in your next maintenance cycle, address maintenance a little bit more fully. But that should be defined for all of the stakeholders in the region, some methodology for feedback from the project to the regional architecture. I hope that address the question that the questioner had provided. Emiliano, did you have another perspective to that?
Emiliano Lopez: For that architecture to be worth anything, it has to be consistent. So understandably so for some agencies that don't have the resources, but if they're not putting in current information if, in fact they're doing systems that need to be somehow fed back into the architecture to say, "Okay, this system is now complete." It may, in fact, end up affecting someone else when they do their system and they're developing and implementing their system based on the notion that this other agency who hasn't provided an update didn't update them to let them know that system was completed. So there needs to be some type of quality assurance and quality control. Again, those are some of the things that we ask you when you develop your maintenance plan are that you take a look at resources and possibly the best way to do that. It may have to be that a consultant, and you may have to pull funding resources that you might have used for something else. I know it's not a popular answer, but it's probably the one I can only offer.
Mac Lister: We have a couple of little comments here, which I thought are real good, so I'll repeat them.
Mac Lister: Comment on the previous question. I think that was on the subregional ATMS system. The comment is: The timing of the subsequent letting of the subregional ATMS projects would likely require another review of the original systems engineering analysis. A tailored systems engineering report would likely require little effort from a large corridor systems engineering report. I think that's a very appropriate and very good answer. When we mentioned project factors regarding the subregional architectures, I'm not sure we mentioned timing. If we didn't, the commenter is very appropriate in saying that.
Mac Lister: A comment on previous question: Agreed, but unfortunately ConOps will not typically be funded until the project is in the TIP or the STIP. I can't remember exactly what the context of that was, so I don't have a response to that. How far in advance of actual detailed design construction of a project does it make sense to start down the SE process, ConOps high level requirements, etc? That really is so project dependent on the scope and size of the project. We talked about one of the concepts that we promote is taking bite-sized pieces. In other words, try to do the project in smaller increments, if possible, because change is inevitable. The longer the cycle between the ConOps and the requirements definitions and the actual implementation, the longer that cycle is, the more change will be required. So we're suggesting bite-sized pieces so that you pick up some of those changes in some of the other increments down the road. So I guess I can't really give you a good answer to how far in advance. There are so many project dependencies on that. Emiliano, did you want to add to that?
Emiliano Lopez: I guess I would probably start right there in preliminary engineering, when you're in the STIP/TIP. At some point, that is handed down to a group in your state or local agency to start developing the project, the specifications and all the other traditional things that you do to get that project up for bid. At that time, when you're doing your preliminary engineering is probably the time to start your system engineering process of evaluations and documentation and all that other good system engineering stuff. I guess the point being, all of your normal process that you would use and that incorporate the system engineering activities and processes that you would need for that particular project.
