The next stage of the systems engineering process, stage 11 is testing evaluation. You'll see on the next slide that actually testing evaluation occurs throughout the system's life cycle. But we'll emphasize it here because it plays a very important role at this end part. First of all, test and evaluation, don't think of those as one term. Number one, we are testing, we are putting the system in some type of infrastructure to test it. We're collecting data, after we collect the data we evaluate. So it's like a two stage process, it's not just one term. We test and we evaluate. So what we're doing is we're testing it, it's a process, or we're collecting data. And then once we collect that data, we analyze it and compare the output against the requirements and the specifications. We want to make sure that the system and its components meet the requirements. We are collecting that data, we are analyzing that data to prove that fact. If we do not do that, we will not know does it meet the requirements, as well as we will not be sure that it's been integrated properly. Furthering the discussion on stage 11 of the systems engineering process, test and evaluation. What you'll want to see is number one is that test and evaluation occurs throughout the system's life-cycle. I have picked this system is engineering V-rubric just to discuss what I mean by this. First of all, we talked about requirements and developing functions. When you do that, you're going to be holding some type of test event. Don't think about it in terms of hardware, but there's some event where you're testing and you're collecting data to make sure you understand what goals you're trying to achieve, what type of functions you're trying to achieve. It's later on when we start realizing that functional to design, where we start getting into more classical test and evaluation infrastructures. We talked earlier about prototyping, we're trying to understand concepts and get concepts in place. But we may have test and evaluation processes here where we're testing early on concepts. Eventually when we're down to the component level, we are looking at very detailed testing on different components to make sure that they functionally work and physically work. Eventually, through the process, we talked about integration, you are bringing these elements together and there will be a test and evaluation. All those different elements that you bring together in integration. Next of all, you will have this complete integrated system, where you'll start doing tests and evaluation on an engineering level. And eventually the test and evaluation will take place with the operational stakeholder. In this case, it can be administrators, clinicians, as well as others in healthcare. What's interesting to see in the systems engineering process, on the left hand side, you're creating measures and metrics, I mentioned those in terms of requirements. What you're doing on the right side of the system engineering V is verifying and validating that in fact you have met those measures and metrics and requirements. So at an engineering level, at a system level, as well as at an operational level. You are confirming, through test and evaluation, that you did in fact meet those requirements. Next stage of the systems engineering process, stage 12, is production and deployment. First of all, again, two terms here, production and deployment. Number one, what we're doing is, we're taking something that we have designed, integrated and built, and we're going to produce it. We don't know if we're going to build 20 of them or 100 of them, hopefully early on in the process, you gotta figure that out. But you're going to reproduce the item that you have developed. In some cases, like a ventilator system, that's more intuitive. But when you get to the idea of like a system of system or an enterprise system, that's more logistically and conceptually difficult to imagine. But again, you may be producing that. Also the idea of a deployment, what we're doing is we're taking those units that have been produced and we're putting them in some type of unit, like an ICU or a hospital or somewhere. We need to understand both of these in order to do them well. We have taken the product, we have designed it, we have integrated it, we have tested it, now we're going to produce it. Recognize though, when you produce these things, there will also be test and evaluation on these produced items. When you deploy as well you put things out in a hospital, in an ICU. You also need to do test and evaluation to make sure when it's deployed, it's going to function properly. But we're at the stages at the very end of the systems engineering process, where we hopefully have been to the point where we have verified and validated that the system that we have made is good. We are producing it now and we're deploying it and there is all kinds of logistics associated with it. As well as continued monitoring, what has been deployed to make sure that we've done it correctly. Elaborating a bit more on stage 12 of the systems engineering process, production and deployment, here are a few items that help illuminate and elaborate on what goes on here. Number one, the knowledge base. Recognize we have been working with a series of designers, operators, and in some cases testers. We need to make sure that we have the right knowledge base for those that can produce and deploy. In most cases, it's a different knowledge base as well as experience base. There are people there that produce things as well as deploy things. Take advantage of that knowledge base. Do not try to take your design engineers or your design folks and come up with some type of production or deployment approach. Take advantage of the existing knowledge base. Also, there's an organizational element there. Health from a systems engineering perspective, do you organize so that you are most efficient to design a system integrated and then transition it to production as well as deployment. Also, when you think about deployment, there is a series of infrastructure. There's an environment that you're putting forward, that is actually going to house the system of system or enterprise system that you've put forward. Be very clear on what that infrastructure is and how you're going to install things. How are you going to deploy things, what's the timing, what are the logistics? Also, when you go to produce as well as to deploy, there are standards, there's regulations that you need to adhere to. Please make sure that you are aware of what those standards, what those guidelines are, what those policies are. So that you do not fall into any trap, that when you actually reach that point of deploying, that you're not allowed to make sure that you have taken that into account and you're have a seamless process for production and deployment.
