WEBVTT 00:00:13.887 --> 00:00:17.717 I'm with the science and Technology directorate. 00:00:17.717 --> 00:00:19.977 We will begin shortly. 00:00:24.777 --> 00:00:28.187 Three to five minutes to let the attendees roll in. 00:00:28.187 --> 00:00:31.367 But thanks for coming up. This is a. 00:00:31.367 --> 00:00:34.527 very important part of the application, and I would actually say. 00:00:34.527 --> 00:00:36.647 the most important part. It's a. 00:00:36.997 --> 00:00:40.097 Of your project proposal. So let's just. 00:00:40.097 --> 00:00:43.377 give it a second and we'll get started soon. 00:00:43.457 --> 00:00:53.457 00:00:53.457 --> 00:01:03.457 00:01:03.457 --> 00:01:13.457 00:01:13.457 --> 00:01:23.457 00:01:23.457 --> 00:01:33.457 00:01:33.457 --> 00:01:43.457 00:01:43.457 --> 00:01:53.457 00:01:53.457 --> 00:02:03.457 00:02:03.457 --> 00:02:13.457 00:02:13.457 --> 00:02:23.457 00:02:23.457 --> 00:02:33.457 00:02:42.877 --> 00:02:43.457 00:02:45.607 --> 00:02:49.147 OK, I see we have some more attendees rolling in soum. 00:02:49.147 --> 00:02:49.147 00:02:50.037 --> 00:02:53.897 My name is Hector Tejeda I'm a program analyst but 00:02:53.897 --> 00:02:57.087 targeted violence and terrorism prevention grant program with. 00:02:57.087 --> 00:03:00.107 the Office of Sensor Prevention. 00:03:00.107 --> 00:03:00.947 and partnerships. 00:03:01.567 --> 00:03:02.287 Uhm? 00:03:03.097 --> 00:03:06.477 We've been posting couple of webinars to help Grant. 00:03:06.477 --> 00:03:09.507 applicants. Make sure they're better prepared. 00:03:09.507 --> 00:03:12.877 to submit high, high quality. 00:03:12.877 --> 00:03:16.967 applications. So our first webinar we. 00:03:16.967 --> 00:03:20.347 promoted. Attendee grants webinar. 00:03:20.347 --> 00:03:23.487 hich is one of the portals portals you would use me application. 00:03:23.487 --> 00:03:26.847 and we also had a webinar on. 00:03:26.847 --> 00:03:29.867 some of our grantees talking about that application experience. 00:03:29.867 --> 00:03:33.607 some tips and just the history of working in. 00:03:33.607 --> 00:03:33.607 00:03:33.657 --> 00:03:35.637 This type of space, this type of work right? 00:03:36.547 --> 00:03:39.597 So I think this workshop today is going to be 1. 00:03:39.597 --> 00:03:43.297 e of the most important. Actually, because it's a. 00:03:43.297 --> 00:03:43.297 00:03:44.327 --> 00:03:48.457 One of the most important parts that the application proposal. This is where you get. 00:03:51.467 --> 00:03:54.697 And that your project is you. 00:03:54.697 --> 00:03:58.007 a project we want to fund, right? It's has. 00:03:58.007 --> 00:04:01.167 great measures to see top show us what? 00:04:01.167 --> 00:04:03.037 works in this field and what doesn't work. 00:04:03.537 --> 00:04:06.477 And it's. 00:04:07.217 --> 00:04:10.447 It's a project that it's feasible so. 00:04:10.447 --> 00:04:14.167 this is. I suggest everyone really take notes listening. 00:04:14.167 --> 00:04:17.347 into this workshop I'm going to hand it off now. 00:04:17.347 --> 00:04:20.757 to Kathleen. It's different from the science and Technology Directorate. 00:04:20.757 --> 00:04:23.867 want to thank her for doing this presentation? 00:04:23.867 --> 00:04:27.127 because she's on state expert on this and I learned every every time. 00:04:27.127 --> 00:04:30.367 I hear it. I learn more and more it again. So thank. 00:04:30.367 --> 00:04:31.727 you all for attending and. 00:04:33.037 --> 00:04:36.227 Yes, let's listen to this great presentation. Now go ahead. 00:04:36.947 --> 00:04:40.257 Great thank you for that introduction, Hector and. 00:04:40.257 --> 00:04:43.547 thank you for everyone for joining today, the. 00:04:43.547 --> 00:04:46.747 presentation today is going to be on metrics and measures. 00:04:46.747 --> 00:04:50.167 and really looking at performance measurement. 00:04:50.167 --> 00:04:50.167 00:04:50.907 --> 00:04:51.467 Uhm? 00:04:53.387 --> 00:04:56.837 There we go, making sure I can move this correctly in. 00:04:56.837 --> 00:04:59.957 the teams environment so. 00:04:59.957 --> 00:05:03.437 the presentation today is really going to aid you. 00:05:03.437 --> 00:05:06.557 as you're filling in your implementation. 00:05:06.557 --> 00:05:09.817 and measurement plan on the grant application. 00:05:09.817 --> 00:05:12.877 we are going to do some level setting and making. 00:05:12.877 --> 00:05:16.217 sure that everyone understands what different. 00:05:16.217 --> 00:05:19.237 key terms mean and will walk through. 00:05:19.237 --> 00:05:22.257 a few different examples of. 00:05:22.257 --> 00:05:22.257 00:05:23.197 --> 00:05:27.477 Identifying goals, objectives and measures. 00:05:27.477 --> 00:05:27.477 00:05:28.207 --> 00:05:31.607 So the 1st place I'll start is making. 00:05:31.607 --> 00:05:35.027 sure that we understand the difference between performance measurement. 00:05:35.027 --> 00:05:38.127 and performance management. So performance. 00:05:38.127 --> 00:05:41.787 measurement is the process of collecting. 00:05:41.787 --> 00:05:45.147 analyzing and or reporting information. 00:05:45.147 --> 00:05:48.227 about performance of an individual group or. 00:05:48.227 --> 00:05:49.107 organization. 00:05:49.677 --> 00:05:53.697 So this is about studying processes, studying strategies. 00:05:53.697 --> 00:05:56.847 and looking to see whether or not. 00:05:56.847 --> 00:05:59.927 an organization's outputs are in line with what they. 00:05:59.927 --> 00:06:01.407 initially intended. 00:06:02.787 --> 00:06:06.497 Performance management is about using. 00:06:06.497 --> 00:06:09.717 those performance measures that were collected, so using. 00:06:09.717 --> 00:06:11.117 the data collected. 00:06:12.127 --> 00:06:15.237 And in order to inform decisions. 00:06:15.237 --> 00:06:18.597 about how to best achieve the mission and deliver. 00:06:18.597 --> 00:06:20.357 results came. 00:06:21.207 --> 00:06:24.317 So hopefully this will all make. Like I said, a little bit. 00:06:24.317 --> 00:06:28.287 more sense as we dig into all of the key terms so. 00:06:28.287 --> 00:06:31.937 the first question is why do we measure performance? 00:06:31.937 --> 00:06:35.237 Measuring performance allows us. 00:06:35.237 --> 00:06:38.867 to manage our near term efforts in. 00:06:38.867 --> 00:06:42.537 order to achieve our intended long term. 00:06:42.537 --> 00:06:43.577 results. 00:06:44.287 --> 00:06:48.327 Great, especially when we look in the field of targeted. 00:06:48.327 --> 00:06:51.627 violence and terrorism prevention. We are. 00:06:51.627 --> 00:06:54.657 at as you put in your grant so you're going to be taking. 00:06:54.657 --> 00:06:58.307 actions now, right? You're going to be developing trainings. 00:06:58.307 --> 00:07:01.467 developing outreach or intervention programs. 00:07:01.467 --> 00:07:04.647 that are being undertaken today. 00:07:04.647 --> 00:07:07.907 and you're overarching goal. Maybe that. 00:07:07.907 --> 00:07:10.927 these lead to fewer levels of. 00:07:10.927 --> 00:07:13.667 violence in your communities in the future. 00:07:14.207 --> 00:07:17.547 But it's really hard to measure that. 00:07:17.547 --> 00:07:20.587 today, because we're talking about a future outcome. 00:07:20.587 --> 00:07:23.767 So again, why do we collect? 00:07:23.767 --> 00:07:26.867 data and measure now? Because it helps us manage our near. 00:07:26.867 --> 00:07:30.497 term efforts to make sure that we're getting our long term results. 00:07:30.497 --> 00:07:30.497 00:07:31.157 --> 00:07:35.167 It allows us to help guide our progress. 00:07:35.167 --> 00:07:36.687 and manage our work. 00:07:37.847 --> 00:07:41.347 It allows you to really importantly to. 00:07:41.347 --> 00:07:45.087 document any successes that you're having, which. 00:07:45.087 --> 00:07:48.247 could be useful to your organization as you apply. 00:07:48.247 --> 00:07:51.327 for other grant opportunities in the future as well to show that. 00:07:51.327 --> 00:07:54.907 you have documented success in past grants. 00:07:54.907 --> 00:07:54.907 00:07:56.347 --> 00:07:59.517 And it can help inform not only your decisions. 00:07:59.517 --> 00:08:02.557 but the decisions of DHS. 00:08:02.557 --> 00:08:05.877 in the types of activities to fund in the future. 00:08:05.877 --> 00:08:05.877 00:08:06.897 --> 00:08:08.627 One other thing that I'll note. 00:08:09.337 --> 00:08:12.697 Is that another reason that performance measurement is so important? 00:08:12.697 --> 00:08:15.857 in this field is because when we're talking about. 00:08:15.857 --> 00:08:19.477 violence prevention of any type, but especially the prevention? 00:08:19.477 --> 00:08:22.797 they've targeted violence and terrorism we really. 00:08:22.797 --> 00:08:25.857 want to make sure that our programs do. 00:08:25.857 --> 00:08:28.987 no harm, right? We don't, yeah. 00:08:28.987 --> 00:08:32.237 we want to make sure that they're achieving their goals with limited. 00:08:32.237 --> 00:08:35.447 unintended consequences and measuring. 00:08:35.447 --> 00:08:38.457 performance of our near term efforts helps. 00:08:38.457 --> 00:08:39.687 us identify. 00:08:39.997 --> 00:08:43.507 What works, what doesn't, and what's promising. 00:08:43.507 --> 00:08:43.507 00:08:47.387 --> 00:08:50.537 So the first step in. 00:08:50.537 --> 00:08:54.777 performance measurement is really creating. 00:08:54.777 --> 00:08:57.137 a performance logic model. 00:08:58.097 --> 00:09:01.737 No, when you create a performance logic model, you actually start. 00:09:01.737 --> 00:09:01.737 00:09:02.797 --> 00:09:05.987 At the end and you. 00:09:05.987 --> 00:09:09.307 ask yourself what impact? 00:09:09.307 --> 00:09:12.627 you want to have, right so? 00:09:12.627 --> 00:09:16.157 if this is successful, what is the change? 00:09:16.157 --> 00:09:19.307 in public condition that you want to? 00:09:19.307 --> 00:09:21.727 come about because of your program? 00:09:22.947 --> 00:09:25.637 And then you start walking that back. 00:09:26.687 --> 00:09:30.187 In order to get that change in public. 00:09:30.187 --> 00:09:33.347 condition, what are the changes? 00:09:33.347 --> 00:09:36.867 in behavior in knowledge and capability? 00:09:36.867 --> 00:09:40.227 What are these outcomes that you would need? 00:09:40.227 --> 00:09:43.577 in order to have that change in condition? 00:09:43.577 --> 00:09:45.227 in order to have that impact? 00:09:46.187 --> 00:09:49.587 What outputs am I? 00:09:49.587 --> 00:09:54.017 going to need in order to get those changes in behavior? 00:09:54.017 --> 00:09:57.167 or capability or skill you? 00:09:57.167 --> 00:10:00.767 t are the products or services? 00:10:00.767 --> 00:10:00.767 00:10:01.907 --> 00:10:04.707 That are going to come out on the tail end. 00:10:05.727 --> 00:10:08.797 And once you've considered that Daniel take another. 00:10:08.797 --> 00:10:11.857 step back and say OK in order to have. 00:10:11.857 --> 00:10:15.077 those outputs, what activities do? 00:10:15.077 --> 00:10:16.497 I need to undertake? 00:10:17.187 --> 00:10:20.327 Great is it training is. 00:10:20.327 --> 00:10:23.387 it community outreach activities? 00:10:23.387 --> 00:10:26.677 Is it train the trainer activities right? What are the activities? 00:10:26.677 --> 00:10:30.167 that I need to undertake in order to get to those outputs that I want? 00:10:30.167 --> 00:10:33.227 And then once you've identified your activities? 00:10:33.227 --> 00:10:33.227 00:10:34.007 --> 00:10:37.217 Finally, you'll consider what the resources. 00:10:37.217 --> 00:10:40.357 or inputs are that you'll need. 00:10:40.357 --> 00:10:40.357 00:10:41.847 --> 00:10:42.697 And this is. 00:10:43.827 --> 00:10:46.997 Really something I would say that you should sit. 00:10:46.997 --> 00:10:50.547 down and try to map out your performance logic model even before you start. 00:10:50.547 --> 00:10:53.807 putting pen to paper on the grant application because. 00:10:53.807 --> 00:10:57.347 if you plan from impact. 00:10:57.347 --> 00:11:00.507 all the way back to resources or inputs. 00:11:00.507 --> 00:11:03.587 it's really going to help you clarify what. 00:11:03.587 --> 00:11:06.667 you request in the grant application and ensure. 00:11:06.667 --> 00:11:08.347 that you have a strong. 00:11:10.147 --> 00:11:14.237 Justification for making that request now. 00:11:14.237 --> 00:11:17.317 the nice thing is that once you have a performance logic model in. 00:11:17.317 --> 00:11:20.977 place, you can use it from left to. 00:11:20.977 --> 00:11:24.317 right from resources, throught impact to. 00:11:24.317 --> 00:11:27.937 ctually manage your activities and ensure. 00:11:27.937 --> 00:11:31.417 that you your resources are being used towards. 00:11:31.417 --> 00:11:35.247 the planned activities that the activities are resulting. 00:11:35.247 --> 00:11:38.397 in the outputs that you wanted right and so. 00:11:38.397 --> 00:11:40.107 you you're able to manage your work. 00:11:40.157 --> 00:11:43.427 And ensure that you're staying on the right path. 00:11:43.427 --> 00:11:46.847 that will eventually lead you to that large impact that. 00:11:46.847 --> 00:11:50.367 you want to have. So the performance logic model is. 00:11:50.367 --> 00:11:52.707 an incredibly powerful tool. 00:11:58.507 --> 00:12:01.657 So let's talk about some. 00:12:01.657 --> 00:12:04.737 key terms. First, we'll talk about. 00:12:04.737 --> 00:12:07.197 goals versus objectives. 00:12:08.117 --> 00:12:10.917 A man on the next slide. I'll provide you some examples. 00:12:11.617 --> 00:12:14.827 So a goal, a goal is a broad. 00:12:14.827 --> 00:12:18.627 brief statement of intent that provides focus. 00:12:18.627 --> 00:12:20.007 and vision. 00:12:21.377 --> 00:12:25.437 Schools tend to be non specific nonmeasurable. 00:12:25.437 --> 00:12:25.437 00:12:26.157 --> 00:12:29.167 And usually cannot be attained solely. 00:12:29.167 --> 00:12:32.587 by the program that you are proposing. 00:12:32.587 --> 00:12:35.727 right? So, as an example ending. 00:12:35.727 --> 00:12:39.147 world is not an attainable goal for a local program. 00:12:39.147 --> 00:12:42.767 right? So again, the goals provide vision. 00:12:42.767 --> 00:12:45.827 and focus Azure plan. 00:12:45.827 --> 00:12:45.827 00:12:47.557 --> 00:12:51.417 Objectives, on the other hand, are realistic targets. 00:12:51.417 --> 00:12:52.967 that your programmer project. 00:12:53.377 --> 00:12:57.257 Uhm, needs to achieve or meet objectives. 00:12:57.257 --> 00:13:00.817 are written in an active tense and use. 00:13:00.817 --> 00:13:03.917 strong verbs like plan right conduct. 00:13:03.917 --> 00:13:05.707 or produce right? 00:13:06.557 --> 00:13:09.717 Objectives will help you focus your program. 00:13:09.717 --> 00:13:10.947 on what matters. 00:13:11.887 --> 00:13:15.237 And when we talk about objectives, we often talk. 00:13:15.237 --> 00:13:17.737 about objectives being smart. 00:13:18.927 --> 00:13:23.177 And smart stands for specific measurable. 00:13:23.177 --> 00:13:26.387 achievable, realistic and. 00:13:26.387 --> 00:13:29.457 time bound. So again, Azure writing your. 00:13:29.457 --> 00:13:33.137 objectives in your implementation and measurement plan. 00:13:33.137 --> 00:13:36.597 We should go through and make sure. 00:13:36.597 --> 00:13:39.807 that they meet each of these criteria so. 00:13:40.967 --> 00:13:44.297 Specific, are they specific to the program? 00:13:44.297 --> 00:13:47.437 that you are undertaking in the Community that you are working? 00:13:47.437 --> 00:13:47.757 in? 00:13:48.637 --> 00:13:51.827 Measurable can we collect? 00:13:51.827 --> 00:13:55.027 data against these objectives to? 00:13:55.027 --> 00:13:57.247 see whether or not they are being met? 00:13:58.817 --> 00:14:02.747 Achievable? Is this something that your program? 00:14:02.747 --> 00:14:05.847 can do in the amount of money? 00:14:05.847 --> 00:14:09.387 and time and other resources that you have? 00:14:09.387 --> 00:14:12.057 right? Is this something that's actually achievable? 00:14:13.147 --> 00:14:16.157 Are they realistic, right? 00:14:16.157 --> 00:14:19.207 So achievable and realistic. Really kind of go hand. 00:14:19.207 --> 00:14:19.667 in hand. 00:14:21.237 --> 00:14:24.997 But is this something that your program? 00:14:24.997 --> 00:14:27.127 will be able to realize? 00:14:28.757 --> 00:14:32.287 And timebound, do you have a set period? 00:14:32.287 --> 00:14:35.817 of time in which she planned to achieve? 00:14:35.817 --> 00:14:38.977 these objectives right? It might be over. 00:14:38.977 --> 00:14:42.197 the course of the grant, or some smaller period within the grant. 00:14:42.197 --> 00:14:45.307 depending on what the objective is. 00:14:45.307 --> 00:14:45.307 00:14:46.857 --> 00:14:47.487 So. 00:14:49.007 --> 00:14:50.657 Or two examples real quick. 00:14:52.537 --> 00:14:55.727 Something like reducing violent extremists? 00:14:55.727 --> 00:14:59.087 recruitment online that is a goal it. 00:14:59.087 --> 00:15:02.207 provides focus. It provides vision. 00:15:02.207 --> 00:15:05.427 but it is unlikely to be achieved. 00:15:05.427 --> 00:15:08.927 through a single program, right? There are just too. 00:15:08.927 --> 00:15:10.767 many other factors in play. 00:15:12.767 --> 00:15:16.027 Similarly, another goal might be to increase community. 00:15:16.027 --> 00:15:19.607 resilience to violent extremists recruitment. 00:15:19.607 --> 00:15:23.487 right again, this is a very good focusing. 00:15:23.487 --> 00:15:26.637 visioning statement for an. 00:15:26.637 --> 00:15:29.847 organization, or at least a focusing. 00:15:29.847 --> 00:15:30.267 visioning. 00:15:30.327 --> 00:15:33.537 From I idea that maybe needs to be. 00:15:33.537 --> 00:15:33.537 00:15:34.497 --> 00:15:36.627 Wordsmith to little bit for an organization. 00:15:37.197 --> 00:15:38.547 Uhm, but. 00:15:39.277 --> 00:15:42.297 It doesn't meet those requirements of being smart. 00:15:42.297 --> 00:15:42.297 00:15:43.127 --> 00:15:43.407 Right? 00:15:44.667 --> 00:15:47.827 On the other hand, we have two examples of objectives. 00:15:47.827 --> 00:15:49.707 here, the 1st. 00:15:50.607 --> 00:15:53.767 Increased participation in an. 00:15:53.767 --> 00:15:57.367 organization called Youth Resilience XYZ by. 00:15:57.367 --> 00:16:00.427 10% by September 20. 00:16:00.427 --> 00:16:03.597 2. Yes no. The timing works. 00:16:03.597 --> 00:16:03.597 00:16:05.287 --> 00:16:05.877 Uhm? 00:16:06.707 --> 00:16:09.737 So this is a smart. 00:16:09.737 --> 00:16:12.117 objective, right? It's very specific. 00:16:13.387 --> 00:16:16.397 We want we know the organization we're dealing with, we know. 00:16:16.397 --> 00:16:19.577 the amount that we want to increase participation by. 00:16:19.577 --> 00:16:23.437 We know that time period. We want to do so. 00:16:23.437 --> 00:16:27.007 we're good there. It's measurable. We. 00:16:27.007 --> 00:16:30.077 have an understanding of to measure how many people. 00:16:30.077 --> 00:16:33.437 are engaged in an organization, right? So we can collect. 00:16:33.437 --> 00:16:34.797 data against us. 00:16:35.657 --> 00:16:38.747 Uhm, it's achievable 10%. 00:16:38.747 --> 00:16:42.167 is not a wildly unrealistic number. 00:16:42.167 --> 00:16:45.527 by which to increase an organization size. 00:16:45.527 --> 00:16:47.647 over a three quarter period. 00:16:48.827 --> 00:16:51.247 Uhm, smart, realistic again. 00:16:52.857 --> 00:16:54.397 The it's. 00:16:55.407 --> 00:16:58.967 We've scaled this down enough that this is a realistic. 00:16:58.967 --> 00:17:02.017 objective for our organization to meet. 00:17:02.017 --> 00:17:05.157 up, and the last one is time bound, so. 00:17:05.157 --> 00:17:08.737 we have a time period September 2022, when. 00:17:08.737 --> 00:17:12.217 e know we want to do the final measurement on this. So that would be considered. 00:17:12.217 --> 00:17:13.457 a smart objective. 00:17:14.677 --> 00:17:17.717 Similarly with our second objective listed here, we might also. 00:17:17.717 --> 00:17:21.347 look at increase awareness of radicalization. 00:17:21.347 --> 00:17:22.097 online. 00:17:23.067 --> 00:17:26.197 By increasing ad views on Facebook by 20. 00:17:26.197 --> 00:17:27.997 this quarter. 00:17:29.027 --> 00:17:32.147 OK, so again this is specific we. 00:17:32.147 --> 00:17:35.627 didn't stop at increase awareness of radicalization. 00:17:35.627 --> 00:17:38.747 online that would not be very specific. 00:17:38.747 --> 00:17:42.047 but we said how we're going to do it, that's. 00:17:42.047 --> 00:17:44.307 what allows us to get to this specificity. 00:17:45.687 --> 00:17:49.067 We can measure this, right? We can look at. 00:17:49.067 --> 00:17:52.107 click through information from Facebook. So we. 00:17:52.107 --> 00:17:55.747 an collect data against it is achievable. 00:17:55.747 --> 00:17:58.757 and realistic right where you. 00:17:58.757 --> 00:18:01.867 king at a 25% increase. 00:18:01.867 --> 00:18:05.157 in AD views? Sure Bar campaign might have to be better. 00:18:05.157 --> 00:18:05.157 00:18:06.767 --> 00:18:09.897 Then the average but this. 00:18:09.897 --> 00:18:12.937 is something that can be attained and we. 00:18:12.937 --> 00:18:15.977 have a time bound. We are looking at it in a one. 00:18:15.977 --> 00:18:17.057 quarter period. 00:18:18.357 --> 00:18:21.397 So keep these in mind as you're writing your goals and. 00:18:21.397 --> 00:18:24.777 objectives, and whenever you're writing an objective. 00:18:24.777 --> 00:18:28.047 always walk through and make sure that it meets. 00:18:28.047 --> 00:18:30.437 the smart objectives. 00:18:33.977 --> 00:18:36.987 For some additional terminology, next we're going. 00:18:36.987 --> 00:18:40.267 to look at impact long term outcomes. 00:18:40.267 --> 00:18:42.367 and short term outcomes. 00:18:43.107 --> 00:18:46.227 So when we talk about impact, we're talking. 00:18:46.227 --> 00:18:49.327 about a change in the public condition. 00:18:49.327 --> 00:18:52.597 right? This is the ultimate, so what? 00:18:52.597 --> 00:18:56.447 Why does your organization exist? 00:18:56.447 --> 00:18:59.667 right? Why are you doing? 00:18:59.667 --> 00:19:00.287 what you do? 00:19:01.117 --> 00:19:03.677 Sorry this is what you want to bring about. 00:19:05.857 --> 00:19:08.277 For a long term outcome. 00:19:09.447 --> 00:19:12.277 We're talking about a change in behavior. 00:19:13.127 --> 00:19:16.547 Or a change in practice so. 00:19:16.547 --> 00:19:19.737 this might cause a change in operations. 00:19:19.737 --> 00:19:23.607 A change in results. It's not necessarily. 00:19:23.607 --> 00:19:26.887 going to get us to that big public condition change. 00:19:26.887 --> 00:19:30.847 but it leads to behavioral change among the population. 00:19:30.847 --> 00:19:32.487 which is really important. 00:19:35.127 --> 00:19:38.287 And then we might look at shorter term outcomes. 00:19:38.287 --> 00:19:41.407 Shorter term outcomes are a change in. 00:19:41.407 --> 00:19:43.407 ill, knowledge, or capability. 00:19:44.447 --> 00:19:47.587 These are competencies that. 00:19:47.587 --> 00:19:51.557 might build towards longer term outcomes there. 00:19:51.557 --> 00:19:55.027 the creation of a new skill or new capability. 00:19:55.027 --> 00:19:58.147 and short term outcomes have value. 00:19:58.147 --> 00:20:01.287 in and of themselves, but they're. 00:20:01.287 --> 00:20:05.447 not necessarily the reason that. 00:20:05.447 --> 00:20:05.447 00:20:06.167 --> 00:20:09.317 The organization exists. 00:20:09.317 --> 00:20:12.437 or the reason why the grant would be. 00:20:12.437 --> 00:20:15.447 awarded to an organization, right? But they do still. 00:20:15.447 --> 00:20:19.517 have value to the organization. 00:20:19.517 --> 00:20:19.517 00:20:22.237 --> 00:20:23.397 So as an example. 00:20:25.517 --> 00:20:28.547 Uhm, a change in public condition that we. 00:20:28.547 --> 00:20:31.397 might want to bring about might be something like. 00:20:31.927 --> 00:20:34.997 Uhm, reduced acts of targeted violence in. 00:20:34.997 --> 00:20:36.017 Washington DC. 00:20:36.997 --> 00:20:40.297 Right, that is you. What we would want. 00:20:40.297 --> 00:20:43.837 to change that's happening in the world another. 00:20:43.837 --> 00:20:47.137 one might be less disinformation. 00:20:47.137 --> 00:20:49.617 Being shared online, right? These are. 00:20:50.027 --> 00:20:53.957 Uhm, big lofty changes. 00:20:53.957 --> 00:20:57.357 that we want to bring about. These are the reasons that we're putting. 00:20:57.357 --> 00:20:59.017 in these grant applications. 00:21:01.237 --> 00:21:04.757 A long term outcome or a change in behavior. 00:21:04.757 --> 00:21:07.587 or practice might be something like. 00:21:07.697 --> 00:21:10.707 Uhm, participants in a. 00:21:10.707 --> 00:21:14.267 program right program ABC improve their. 00:21:14.267 --> 00:21:17.567 protective factors are increased their results. 00:21:17.567 --> 00:21:19.877 right? That's important. 00:21:20.537 --> 00:21:23.557 That long term outcome helps you get. 00:21:23.557 --> 00:21:26.717 to the eventual impact that you want to. 00:21:26.717 --> 00:21:28.467 be at right. 00:21:28.867 --> 00:21:32.107 Uhm, but it's also easier to measure and you have a little. 00:21:32.107 --> 00:21:33.347 bit more control over it. 00:21:35.347 --> 00:21:38.617 Short term outcomes. Again, the change and skill knowledge. 00:21:38.617 --> 00:21:41.827 or capability might be something like. 00:21:41.827 --> 00:21:45.347 students in XYZ School District. 00:21:45.347 --> 00:21:48.647 are able to identify misinformation. 00:21:48.647 --> 00:21:51.907 online at higher rates than the general public, right? 00:21:51.907 --> 00:21:55.307 So the students that went through this program have. 00:21:55.307 --> 00:21:58.387 had an increase in skill right there now. 00:21:58.387 --> 00:22:01.807 able to identify misinformation. 00:22:01.807 --> 00:22:05.527 online that is important, right? That is worthwhile. 00:22:05.527 --> 00:22:05.527 00:22:06.957 --> 00:22:10.377 But it's not your overall reason. 00:22:10.377 --> 00:22:14.437 Your overall change in public condition that you want of less dis information being shared. 00:22:14.437 --> 00:22:17.517 online, but it's a building block, right? It's? 00:22:17.517 --> 00:22:20.907 a building block that will eventually help you get to. 00:22:20.907 --> 00:22:24.037 your long term impact, and that is always how. 00:22:24.037 --> 00:22:27.417 they should flow, right? Your short term outcomes your. 00:22:27.417 --> 00:22:30.497 long term outcomes should all be building. 00:22:30.497 --> 00:22:33.417 blocks that get you too. 00:22:34.087 --> 00:22:37.367 Uhm, yeah, or that lead eventually to the. 00:22:37.367 --> 00:22:40.587 impact that your organization wants to have. 00:22:40.587 --> 00:22:40.587 00:22:44.967 --> 00:22:48.117 Now let me set expectations a little bit here. 00:22:48.117 --> 00:22:48.117 00:22:48.767 --> 00:22:51.837 Uhm impacts are really hard to measure. 00:22:51.837 --> 00:22:55.237 The outcomes are hard to measure as well. 00:22:55.237 --> 00:22:58.297 There's typically a lot that happens outside of. 00:22:58.297 --> 00:23:01.457 your organization's control, even when. 00:23:01.457 --> 00:23:05.277 ooking at a long or short term outcome. 00:23:05.277 --> 00:23:05.277 00:23:05.977 --> 00:23:09.077 But just because they're hard to measure and. 00:23:09.077 --> 00:23:12.777 there's stuff going on, that's outside of your control that impact. 00:23:12.777 --> 00:23:15.837 Maybe I should pick a different word there, but that. 00:23:15.837 --> 00:23:18.997 affect whether or not people are. 00:23:18.997 --> 00:23:22.197 reaching those outcomes. It doesn't mean that we shouldn't. 00:23:22.197 --> 00:23:25.477 stop and measure them, right? There's always indicators. 00:23:25.477 --> 00:23:28.797 that you can and should look for in order to get a sense. 00:23:28.797 --> 00:23:33.037 of whether or not the investment. 00:23:33.037 --> 00:23:36.597 that you've put in has made a difference, right? 00:23:36.597 --> 00:23:36.597 00:23:37.167 --> 00:23:41.007 But while you're doing that, you should be considering. 00:23:41.007 --> 00:23:44.667 what's happening that's outside your control. That might influence the results. 00:23:44.667 --> 00:23:44.667 00:23:45.337 --> 00:23:48.977 Uhm, and use all of that information. 00:23:48.977 --> 00:23:52.517 to plan your next steps. 00:23:52.517 --> 00:23:55.657 and to interpret the results correctly. 00:23:55.657 --> 00:23:57.197 right? So? 00:23:57.917 --> 00:24:01.427 The awards that were given and. 00:24:01.427 --> 00:24:02.587 2019. 00:24:03.237 --> 00:24:07.027 Uhm, we're probably impacted by Covid, right, and? 00:24:07.317 --> 00:24:10.377 Uhm, the pandemic protocols and. 00:24:10.377 --> 00:24:13.537 people not being able to meet in person anymore. 00:24:13.537 --> 00:24:16.707 So when they looked at their results. 00:24:16.707 --> 00:24:19.777 when they looked at their measurements to see if they admit their objectives. 00:24:19.777 --> 00:24:23.257 or not, they these organizations needed to consider. 00:24:23.257 --> 00:24:26.677 what was happening outside of their control, right? What was happening? 00:24:26.677 --> 00:24:29.777 in the world because of the pandemic? And that's. 00:24:29.777 --> 00:24:33.227 extreme example, but it's still an important one. 00:24:33.227 --> 00:24:33.227 00:24:33.287 --> 00:24:36.327 And they needed to consider that so that. 00:24:36.327 --> 00:24:39.147 they could correctly interpret their results that they were getting. 00:24:42.557 --> 00:24:43.907 Another thing to consider. 00:24:44.587 --> 00:24:46.017 Is that it's? 00:24:46.897 --> 00:24:50.297 Usually the case that one measure is not enough. 00:24:50.297 --> 00:24:50.297 00:24:51.207 --> 00:24:54.877 Right in order to understand outcomes. 00:24:54.877 --> 00:24:58.577 you typically need to look at multiple different data points. 00:24:58.577 --> 00:25:01.687 and kind of use all of that information to almost like. 00:25:01.687 --> 00:25:03.037 in triangulate. 00:25:03.777 --> 00:25:06.897 Uhm, what the actual result? 00:25:06.897 --> 00:25:09.977 is focusing on only one measure. 00:25:09.977 --> 00:25:13.777 might give you a distorted view of. 00:25:13.777 --> 00:25:17.597 what your results are. So whenever. 00:25:17.597 --> 00:25:20.817 you can think of more than one. 00:25:20.817 --> 00:25:24.717 measure for an outcome that's. 00:25:24.717 --> 00:25:27.757 a really great thing, so consider that too as. 00:25:27.757 --> 00:25:29.577 you're building out your I MPs. 00:25:32.667 --> 00:25:36.227 Alright, so now I'm gonna walk you through 2 examples the. 00:25:36.227 --> 00:25:39.417 first example is kind of going to be a universal. 00:25:39.417 --> 00:25:42.677 one about traffic and then we'll. 00:25:46.137 --> 00:25:48.297 28 errors and prevention case. 00:25:49.927 --> 00:25:53.337 So let's say that we are running a program. 00:25:53.337 --> 00:25:56.807 that's overarching. Goal is to improve. 00:25:56.807 --> 00:25:59.777 driver and pedestrian safety on Main Street. 00:26:01.987 --> 00:26:05.157 One of their objectives might be. 00:26:05.157 --> 00:26:08.417 that they would like the average vehicle speed on. 00:26:08.417 --> 00:26:11.557 Main Street to be 35 miles per. 00:26:11.557 --> 00:26:14.197 s quarter. 00:26:17.877 --> 00:26:21.247 And so the different activities that they'll undertake. 00:26:21.247 --> 00:26:24.607 and attempting to reach this objective they. 00:26:24.607 --> 00:26:27.687 might have public awareness campaigns on radio. 00:26:27.687 --> 00:26:30.987 TV, billboards. Kind of any way that they can think of to reach. 00:26:30.987 --> 00:26:34.187 people who drive on Main Street. And they'll also. 00:26:34.187 --> 00:26:37.427 put out increased speed displays on the road. So I think. 00:26:37.427 --> 00:26:40.467 e signs that when you drive past them, they tell you how. 00:26:40.467 --> 00:26:41.867 how fast you're going, right? 00:26:45.017 --> 00:26:48.187 And the impact of. 00:26:48.187 --> 00:26:49.247 these. 00:26:50.477 --> 00:26:53.927 Activities right the impact that they want to have is fewer. 00:26:53.927 --> 00:26:56.947 accidents and vehicular deaths OK? 00:26:56.947 --> 00:26:56.947 00:26:59.827 --> 00:27:03.757 And a long term outcome. 00:27:03.757 --> 00:27:07.037 would be decreased average speed on. 00:27:07.037 --> 00:27:10.697 the roads. So remember long term outcome change. 00:27:10.697 --> 00:27:13.737 in behavior right? So decreased average. 00:27:13.737 --> 00:27:17.257 speed is a change in behavior of the drivers. 00:27:17.257 --> 00:27:20.597 and that outcome should be a building. 00:27:20.597 --> 00:27:24.097 block to get us to our impact of fewer accidents. 00:27:24.097 --> 00:27:25.597 and vehicular deaths. 00:27:27.217 --> 00:27:30.527 And a short term outcome might be increased. 00:27:30.527 --> 00:27:33.707 awareness amongst drivers of the importance of adhering. 00:27:33.707 --> 00:27:36.767 to the speed limit. Again the short term. 00:27:36.767 --> 00:27:39.787 outcome provides the building block to. 00:27:39.787 --> 00:27:43.127 our long term outcome, which provides a building block. 00:27:43.127 --> 00:27:44.827 to our impact. 00:27:46.637 --> 00:27:47.887 So with this example. 00:27:49.237 --> 00:27:53.047 We might look at measurement in a few different ways, so. 00:27:53.047 --> 00:27:56.127 one thing we could do if we want to measure the. 00:27:56.127 --> 00:27:59.367 increased awareness of the need to adhere to speed. 00:27:59.367 --> 00:28:02.627 limits. We might do community surveys. 00:28:02.627 --> 00:28:04.557 right? We might look at community surveys. 00:28:04.977 --> 00:28:08.547 Uhm pre our public awareness campaign. 00:28:08.547 --> 00:28:11.817 and post or public awareness campaign and will come back. 00:28:11.817 --> 00:28:15.237 to how to do pre post testing in a few minutes and. 00:28:15.237 --> 00:28:19.237 that will provide us some information on how awareness. 00:28:19.237 --> 00:28:21.597 has changed amongst the community. 00:28:24.797 --> 00:28:27.837 If we want to look at the average speed on a. 00:28:27.837 --> 00:28:30.937 specific stretch of road, right, this is a way that we might want. 00:28:30.937 --> 00:28:33.967 to measure our long term outcome of decreased average speed. 00:28:33.967 --> 00:28:37.487 on streets. We might collect data from. 00:28:37.487 --> 00:28:40.407 the speed displays on the road now. 00:28:43.027 --> 00:28:46.077 These are just one measure, right? You might be. 00:28:46.077 --> 00:28:49.497 able to think of others. We might pull information. 00:28:49.497 --> 00:28:52.767 om traffic tickets as a measure for average. 00:28:52.767 --> 00:28:54.877 e speed on a specific structure Rd. 00:28:56.667 --> 00:28:59.997 You might look at the. 00:28:59.997 --> 00:29:03.117 number of cars that have run red lights right because? 00:29:03.117 --> 00:29:06.347 that you might be tide into the speed. 00:29:06.347 --> 00:29:09.457 that they're going on the street and their inability to stop right so? 00:29:09.457 --> 00:29:13.017 there might be multiple measures that you want to collect. 00:29:13.017 --> 00:29:13.017 00:29:13.967 --> 00:29:16.997 That will help you identify. 00:29:16.997 --> 00:29:20.057 what your results are in terms of looking at those long. 00:29:20.057 --> 00:29:21.597 term and short term outcomes. 00:29:24.427 --> 00:29:27.477 Now to move into a more terrorism prevention. 00:29:27.477 --> 00:29:30.777 example, let's say that you have an organization. 00:29:30.777 --> 00:29:34.217 whose goal is to increase community. 00:29:34.217 --> 00:29:37.417 resilience to violent extremism again. 00:29:37.417 --> 00:29:37.417 00:29:38.427 --> 00:29:41.147 Broad, not smart, right? 00:29:42.827 --> 00:29:46.297 And so that organization. 00:29:46.297 --> 00:29:49.657 may have under its goal is 1 objective. 00:29:49.657 --> 00:29:49.657 00:29:51.517 --> 00:29:54.807 To increase youth membership in. 00:29:54.807 --> 00:29:58.067 ABC, which is a Pro social organization. 00:29:58.067 --> 00:30:01.547 by 25% / a two year period. 00:30:01.547 --> 00:30:03.027 Again this objective. 00:30:03.857 --> 00:30:05.527 It is a smart objective. 00:30:07.817 --> 00:30:11.267 And so the activities that you might undertake are things. 00:30:11.267 --> 00:30:12.787 like. 00:30:13.747 --> 00:30:17.127 Establishing, staffing, and recruiting for the organization. 00:30:17.127 --> 00:30:20.367 Holding regular events to keep youth engaged right. 00:30:20.367 --> 00:30:20.367 00:30:22.237 --> 00:30:25.617 Your hope is that the impact. 00:30:25.617 --> 00:30:28.767 of this organization that impact of having greater. 00:30:28.767 --> 00:30:32.247 Community resilience to violent extremism, is that fewer. 00:30:32.247 --> 00:30:36.227 individuals will join violent extremist organizations. 00:30:36.227 --> 00:30:36.227 00:30:38.127 --> 00:30:41.177 And so long term outcome that. 00:30:41.177 --> 00:30:44.317 provides a building block to that impact is that. 00:30:44.317 --> 00:30:47.637 youth engaged with your organization ABC. 00:30:47.637 --> 00:30:50.817 build protective factors that increase. 00:30:50.817 --> 00:30:54.377 their resilience, right? So this becomes. 00:30:54.377 --> 00:30:57.617 a change in their behavior. 00:30:57.617 --> 00:30:57.617 00:30:59.347 --> 00:31:02.557 Short term outcome again that provides the building block. 00:31:02.557 --> 00:31:06.077 to that longer term outcome is that. 00:31:06.077 --> 00:31:09.117 sorry youth engagement with ABC organization is. 00:31:09.117 --> 00:31:10.457 sustained overtime. 00:31:12.947 --> 00:31:13.437 So. 00:31:14.317 --> 00:31:17.657 There might be a few ways of looking at measurement for this. 00:31:17.657 --> 00:31:17.657 00:31:19.907 --> 00:31:23.247 The short term outcome right that youth engagement is sustained. 00:31:23.247 --> 00:31:23.247 00:31:24.077 --> 00:31:27.487 We might look at a measure. 00:31:27.487 --> 00:31:30.497 that says ABC organization retains 80%. 00:31:30.497 --> 00:31:33.307 of its new members over the life cycle of eligibility. 00:31:34.077 --> 00:31:37.367 In order to measure that, all we need to do is record. 00:31:37.367 --> 00:31:40.867 attendance at regular meetings are very two year period and. 00:31:40.867 --> 00:31:44.157 track how we're doing in terms of retaining. 00:31:44.157 --> 00:31:46.627 members and bringing in new members even. 00:31:49.747 --> 00:31:51.167 If we want to look at. 00:31:51.807 --> 00:31:52.467 Uhm? 00:31:53.277 --> 00:31:56.757 Our long term outcome, a potential measure for that is. 00:31:56.757 --> 00:32:00.177 that participants might see an increased positive youth development. 00:32:00.177 --> 00:32:03.227 score on surveys over. 00:32:03.227 --> 00:32:05.217 the life cycle of the program. 00:32:05.867 --> 00:32:09.177 So again, we would need to do some pre post testing. 00:32:16.837 --> 00:32:20.577 So uhm, I've mentioned pre post testing twice. 00:32:20.577 --> 00:32:20.577 00:32:21.467 --> 00:32:24.387 So let me break down a little bit what I really mean by that. 00:32:25.827 --> 00:32:29.267 Pre post testing is a method. 00:32:29.267 --> 00:32:32.837 that's used to measure change overtime. 00:32:32.837 --> 00:32:32.837 00:32:34.207 --> 00:32:37.347 This is really popular when you're looking at training. 00:32:37.347 --> 00:32:40.077 programs right? And you want to identify something like. 00:32:41.247 --> 00:32:44.387 How your training program impacted what people know. 00:32:44.387 --> 00:32:46.587 right so? 00:32:47.157 --> 00:32:50.467 We call it pre post testing because we do the testing in 2π. 00:32:50.467 --> 00:32:51.487 time periods. 00:32:53.527 --> 00:32:56.547 First, we do the testing in pre time. 00:32:56.547 --> 00:32:59.857 ime period. This is before any. 00:32:59.857 --> 00:33:03.357 intervention or event or training. 00:33:03.357 --> 00:33:05.077 or learning takes place. 00:33:05.907 --> 00:33:09.417 And that helps us establish a baseline for measurement. 00:33:09.417 --> 00:33:11.687 Then we look at post testing. 00:33:12.517 --> 00:33:15.937 And so this is after the intervention has taken place. 00:33:15.937 --> 00:33:19.517 and we identified the new level you. 00:33:19.517 --> 00:33:22.677 rate from the baseline of. 00:33:22.677 --> 00:33:24.897 after the intervention. 00:33:25.787 --> 00:33:29.217 And when we take the difference. 00:33:29.217 --> 00:33:32.297 in those that allows us to measure. 00:33:32.297 --> 00:33:35.987 the change brought about by the intervention. 00:33:35.987 --> 00:33:35.987 00:33:37.007 --> 00:33:40.397 So just some keynotes. 00:33:40.397 --> 00:33:44.677 data collected and reported should be well defined. 00:33:44.677 --> 00:33:44.677 00:33:45.477 --> 00:33:48.507 Should be measurable and it should be clearly. 00:33:48.507 --> 00:33:51.687 linked to the intervention that took place to. 00:33:51.687 --> 00:33:55.117 the training or the program or. 00:33:55.117 --> 00:33:55.807 the event. 00:33:56.737 --> 00:33:59.947 And the population is being tested in the pre and. 00:33:59.947 --> 00:34:01.107 post period. 00:34:02.827 --> 00:34:03.997 There should be similar. 00:34:04.857 --> 00:34:08.007 It's not the exact same during pre. 00:34:08.007 --> 00:34:10.347 pre posting period, so. 00:34:11.477 --> 00:34:14.757 I'll give you an example in just a minute where the populations. 00:34:14.757 --> 00:34:17.967 are exactly the same, but if you think back to our traffic. 00:34:17.967 --> 00:34:21.647 example and we talked about doing community surveys. 00:34:21.647 --> 00:34:25.067 right to see if people's attitudes and 00:34:25.067 --> 00:34:28.367 knowledge about speeding and traffic laws had. 00:34:28.367 --> 00:34:31.487 changed over time, we would want to make sure. Well, we. 00:34:31.487 --> 00:34:34.527 might not be able to provide the surveys to. 00:34:34.527 --> 00:34:37.547 the exact same people in both of those groups we would. 00:34:37.547 --> 00:34:40.717 want to make sure that the populations are similar that we're still looking. 00:34:40.757 --> 00:34:44.107 At drivers. 00:34:44.107 --> 00:34:47.807 aged 16 to 75. 00:34:47.807 --> 00:34:49.877 who live in this community. 00:34:49.927 --> 00:34:53.137 He could have roughly similar. 00:34:53.137 --> 00:34:56.657 demographic makeups once the entire population. 00:34:56.657 --> 00:34:58.277 of the survey comes in. 00:35:02.727 --> 00:35:03.307 So. 00:35:06.687 --> 00:35:09.107 Good pre post tests. 00:35:10.177 --> 00:35:13.337 Come have well defined discrete. 00:35:13.337 --> 00:35:16.417 and measurable variables that are directly linked to the. 00:35:16.417 --> 00:35:17.197 intervention. 00:35:18.267 --> 00:35:19.337 So for example. 00:35:20.187 --> 00:35:23.337 You might want to know the impact of the free lunch program on. 00:35:23.337 --> 00:35:26.907 students access to food at a public school. 00:35:26.907 --> 00:35:26.917 00:35:27.907 --> 00:35:31.087 Uhm, the free lunch program in that case is the intervention. 00:35:31.087 --> 00:35:32.157 that's taking place. 00:35:33.187 --> 00:35:36.597 Uhm, and this is. 00:35:36.597 --> 00:35:40.137 a pretty well defined measurable discrete. 00:35:40.137 --> 00:35:43.997 pre post test. Because students. 00:35:43.997 --> 00:35:47.617 ccess to food is fairly limited when they're out of school. 00:35:47.617 --> 00:35:50.937 right? You've basically got bringing stuff from home the cafeteria. 00:35:50.937 --> 00:35:54.077 and the vending machines. So the. 00:35:54.077 --> 00:35:57.497 free lunch program is likely to have an impact. 00:35:57.497 --> 00:35:57.497 00:35:58.297 --> 00:36:00.087 On their access to food. 00:36:02.347 --> 00:36:05.377 But bad pre post tests tend to. 00:36:05.377 --> 00:36:08.957 be poorly defined and claim effects that could be attributed. 00:36:08.957 --> 00:36:11.187 to a multitude of other factors. 00:36:11.887 --> 00:36:15.027 Uhm or not measurable and may have no direct. 00:36:15.027 --> 00:36:18.467 relation to a specific intervention so. 00:36:18.467 --> 00:36:21.987 if you wanted to look at the effect of protein. 00:36:21.987 --> 00:36:25.127 intake on muscle composition among students. 00:36:25.127 --> 00:36:28.227 right? So your outcome that you're looking at his muscle composition? 00:36:28.227 --> 00:36:31.967 and your intervention is protein intake. 00:36:31.967 --> 00:36:31.967 00:36:32.897 --> 00:36:36.217 I mean, sure, protein intake impacts muscle composition. 00:36:36.217 --> 00:36:39.577 but there's a lot of other stuff that impacts muscle composition. 00:36:39.577 --> 00:36:42.297 Two, most importantly, probably working out. 00:36:42.977 --> 00:36:44.037 Great so. 00:36:46.127 --> 00:36:48.507 We don't have something that has a direct. 00:36:49.727 --> 00:36:53.387 Relationship to the specific intervention. 00:36:53.387 --> 00:36:56.407 right? There's too much else happening that's. 00:36:56.407 --> 00:36:59.627 different across all the different students that might. 00:36:59.627 --> 00:37:03.147 be in your test to really be able to pinpoint. 00:37:03.147 --> 00:37:06.327 and say yes, the difference in muscle composition is based. 00:37:06.327 --> 00:37:08.167 on this protein intake. 00:37:11.397 --> 00:37:13.397 So let's walk through an example. 00:37:14.177 --> 00:37:17.427 Let's say we have an organization whose goal. 00:37:17.427 --> 00:37:19.507 is to increase civic engagement. 00:37:20.877 --> 00:37:24.097 And so their objective is to increase students knowledge of. 00:37:24.097 --> 00:37:27.417 voting in their community by 25% before. 00:37:27.417 --> 00:37:30.577 the primary. So again, if you go back to smart. 00:37:30.577 --> 00:37:33.027 you can apply that to this objective. 00:37:35.837 --> 00:37:38.937 So the measure that we're looking. 00:37:38.937 --> 00:37:43.187 at is we want participants in the vote project. 00:37:43.187 --> 00:37:46.797 to increase their awareness of to access voting resources. 00:37:46.797 --> 00:37:48.297 by 25%. 00:37:51.057 --> 00:37:54.347 So we're going to look at that through pre post testing so. 00:37:54.347 --> 00:37:55.177 what we're going to do? 00:37:56.047 --> 00:37:59.297 Is for the pre tests we wanna understand. 00:37:59.297 --> 00:38:03.237 the students baseline knowledge before entering the program so. 00:38:03.237 --> 00:38:06.617 the first thing that students do when they come to their very first. 00:38:06.617 --> 00:38:08.497 class meeting etc. 00:38:11.317 --> 00:38:14.687 Come to baseline their understanding and awareness. 00:38:14.687 --> 00:38:16.187 of voting in their communities. 00:38:17.277 --> 00:38:20.287 And then, as part of the program, students will. 00:38:20.287 --> 00:38:24.047 be provided lessons on voting eligibility timeframes. 00:38:24.047 --> 00:38:27.247 location of polling stations and how votes are tabulated. 00:38:27.247 --> 00:38:27.247 00:38:29.317 --> 00:38:31.227 After the training is complete. 00:38:32.527 --> 00:38:36.027 The students are given the exact. 00:38:38.747 --> 00:38:41.997 That they took before the training started. 00:38:41.997 --> 00:38:41.997 00:38:43.317 --> 00:38:46.907 And if the average score is higher after. 00:38:46.907 --> 00:38:50.807 the training right after the intervention, then in the first then. 00:38:50.807 --> 00:38:53.987 you can say that the vote project had. 00:38:53.987 --> 00:38:57.527 a positive impact on voting knowledge amongst youth. 00:38:57.527 --> 00:39:00.787 in this community, right? And the difference? 00:39:00.787 --> 00:39:03.647 lets if you hit that 25% mark. 00:39:07.607 --> 00:39:10.907 So just to cycle back, I want to remind. 00:39:10.907 --> 00:39:14.937 everyone of why we measure performance so. 00:39:14.937 --> 00:39:18.367 we measure performance to a identify whether. 00:39:18.367 --> 00:39:21.467 or not we are making progress towards our desired. 00:39:21.467 --> 00:39:24.937 impact. Are we building those building blocks that we need? 00:39:24.937 --> 00:39:26.247 to get to impact? 00:39:27.437 --> 00:39:30.627 We measure performance to identify areas. 00:39:30.627 --> 00:39:33.857 for improvement in the future, including. 00:39:33.857 --> 00:39:36.957 information for DHS that. 00:39:36.957 --> 00:39:40.147 can potentially be provided to other site. 00:39:40.147 --> 00:39:43.587 locations where violence prevention. 00:39:43.587 --> 00:39:46.747 activities are going on and then finally. 00:39:46.747 --> 00:39:50.137 we measure performance in order to continue to grow the knowledge. 00:39:50.137 --> 00:39:53.227 base of what works and targeted violence and. 00:39:53.227 --> 00:39:55.357 terrorism prevention operations. 00:39:56.327 --> 00:39:59.207 Create a meeting. It's very important. 00:39:59.837 --> 00:40:02.957 To ensure that the. 00:40:02.957 --> 00:40:06.317 data you're collecting and the measure. 00:40:06.317 --> 00:40:07.777 measurements are taking. 00:40:08.427 --> 00:40:09.027 Uhm? 00:40:09.967 --> 00:40:13.147 Actually help you walk that path. 00:40:13.147 --> 00:40:13.147 00:40:13.907 --> 00:40:17.227 Towards understanding if you're having the impact that you want to have. 00:40:17.227 --> 00:40:17.227 00:40:19.547 --> 00:40:23.007 I'm just with a few resources. 00:40:23.007 --> 00:40:26.107 listed and these are. 00:40:26.107 --> 00:40:29.627 the same as the information that CP3 pets out, so. 00:40:29.627 --> 00:40:32.807 I'm sure that you'll have this in other ways as well and. 00:40:32.807 --> 00:40:35.287 this presentation. 00:40:35.897 --> 00:40:39.097 Is can be made publicly available so. 00:40:39.097 --> 00:40:40.147 it can be shared. 00:40:40.977 --> 00:40:44.137 By CP3 and I just thank everyone for. 00:40:44.137 --> 00:40:47.287 coming today, and I'm I don't I. 00:40:47.287 --> 00:40:50.777 can't see the chat, but I'll pass it over to Hector to see if there are any questions. 00:40:50.777 --> 00:40:51.837 that I need to answer. 00:40:52.957 --> 00:40:55.737 Yeah, thank you, thanks so much, Kathleen. 00:40:57.977 --> 00:41:00.927 We connect questions coming now. I think people were listening. 00:41:02.577 --> 00:41:05.617 Just a recap, so you said I think one of the. 00:41:05.617 --> 00:41:08.807 questions was is if this can go. 00:41:08.807 --> 00:41:12.177 public and I know we're working on it to get approval to make this go public. 00:41:12.177 --> 00:41:12.177 00:41:12.937 --> 00:41:16.497 So the slide deck is 100% approved for public release. 00:41:16.497 --> 00:41:19.757 and I'm we're. 00:41:19.757 --> 00:41:22.997 on the presentation portion as well, but. 00:41:22.997 --> 00:41:24.167 I think that will be a yes. 00:41:24.597 --> 00:41:28.407 OK, for the recording correct? Yeah OK so. 00:41:28.407 --> 00:41:30.867 o maybe that'll be coming in the upcoming week. 00:41:31.237 --> 00:41:34.407 Uhm, uhm, but thank you so much for the presentation. 00:41:34.407 --> 00:41:37.467 Kathleen again, I want to repeat this is like. 00:41:37.467 --> 00:41:40.847 the most important part of your project proposal. 00:41:40.847 --> 00:41:44.767 Your is your logic model in your implementation. 00:41:44.767 --> 00:41:47.797 measurement plan capturing those measurements so we know that we're going. 00:41:47.797 --> 00:41:50.887 to fund that to the CPT for P3. 00:41:50.887 --> 00:41:53.947 office. No, that's going to fund the program that basically. 00:41:53.947 --> 00:41:56.317 be basically follows a smart mythology. 00:41:56.807 --> 00:41:59.877 Uhm, and it seems realistic. 00:41:59.877 --> 00:42:03.217 right? But if you guys, if anybody in the audience. 00:42:03.217 --> 00:42:07.127 has any questions for Kathleen regarding the. 00:42:07.127 --> 00:42:10.327 presentation you can access now I'm going to look in the chat. 00:42:10.327 --> 00:42:13.757 and I'll forward them to you. Kathleen if they're. 00:42:13.757 --> 00:42:14.407 relevant. 00:42:14.467 --> 00:42:15.117 Uhm? 00:42:15.807 --> 00:42:18.977 I think question from Jordan that I can share this. 00:42:18.977 --> 00:42:19.457 is awesome. 00:42:20.187 --> 00:42:20.727 OK, go ahead. 00:42:21.177 --> 00:42:24.227 So the question is, what is the difference between? 00:42:24.227 --> 00:42:27.327 goals and outcomes? Ideally they should be the same. 00:42:27.327 --> 00:42:30.387 correct? There should be aligned right there should. 00:42:30.387 --> 00:42:33.797 be very closely aligned. You saw in some of my examples. 00:42:33.797 --> 00:42:37.307 they weren't necessarily worded exactly the same, but. 00:42:37.307 --> 00:42:40.447 you should be able to very easily draw thread. 00:42:40.447 --> 00:42:41.347 from one to the other. 00:42:46.487 --> 00:42:49.827 And then I see one in the chat about his pre and post testing. 00:42:49.827 --> 00:42:53.017 always accurate, no right? 00:42:53.017 --> 00:42:56.047 How accurate? Pre and post testing is? 00:42:56.047 --> 00:42:59.647 really is dependent upon what? 00:42:59.647 --> 00:43:03.347 other outside factors can influence. 00:43:03.347 --> 00:43:04.347 your outcome. 00:43:05.047 --> 00:43:06.477 Great so. 00:43:07.107 --> 00:43:07.757 Uhm? 00:43:08.507 --> 00:43:10.297 Pre and post testing can be. 00:43:11.007 --> 00:43:14.347 Fairly accurate for like a closed training. 00:43:14.347 --> 00:43:18.057 where you come in. I give you pre test and. 00:43:18.057 --> 00:43:21.707 d then we do like two or three days of intense training. 00:43:21.707 --> 00:43:25.467 and a subject and then I give you a post test right? 00:43:25.467 --> 00:43:29.277 There's not a lot of time for outside influence. 00:43:29.277 --> 00:43:32.417 over that short period, and so I can. 00:43:32.417 --> 00:43:35.597 pretty accurately say that the change in scores. 00:43:35.597 --> 00:43:38.627 from pre and post are due to the. 00:43:38.627 --> 00:43:40.357 intervention or due to my training. 00:43:41.367 --> 00:43:42.157 But 00:43:42.937 --> 00:43:46.067 If we have something longer term. 00:43:46.067 --> 00:43:49.347 where other factors might influence what's going. 00:43:49.347 --> 00:43:51.617 on too, right like so if I wanna say. 00:43:51.667 --> 00:43:53.177 Hey come. 00:43:53.987 --> 00:43:56.187 Study how. 00:43:56.247 --> 00:43:59.527 Hold on a change in. 00:43:59.527 --> 00:44:02.817 the unemployment rate or a change in. 00:44:02.817 --> 00:44:05.967 minimum wage effects. Unemployment, right? So that's a big one that I'm. 00:44:05.967 --> 00:44:09.087 not necessarily going to touch with a 10 foot pole, but there's a. 00:44:09.087 --> 00:44:12.187 lot of other factors. 00:44:12.187 --> 00:44:15.487 that influence the unemployment rate, including timing. 00:44:15.487 --> 00:44:17.317 If I asked that question. 00:44:19.507 --> 00:44:23.107 In January of 2020, and I studied throughout 2020. 00:44:23.107 --> 00:44:23.107 00:44:24.757 --> 00:44:27.827 My results would be so skewed because the unemployment. 00:44:27.827 --> 00:44:31.287 rate shot up because of Covid right? Whereas if I asked it in a. 00:44:31.287 --> 00:44:34.857 different time period, I might get less skewed results. 00:44:34.857 --> 00:44:38.147 So whenever you're doing pre post testing you have. 00:44:38.147 --> 00:44:41.567 to consider those outside factors that. 00:44:41.567 --> 00:44:44.927 are also impacting that. 00:44:44.927 --> 00:44:48.447 outcome that you're looking at right so? 00:44:48.447 --> 00:44:51.187 please the great question, but please do keep that in mind. 00:44:53.997 --> 00:44:55.947 I think we have a meeting. 00:44:57.517 --> 00:45:00.547 Yeah question here. I mean I think you touched upon it, but. 00:45:00.547 --> 00:45:04.177 if you can maybe reiterate the point on. 00:45:04.177 --> 00:45:06.387 in a few words. 00:45:06.897 --> 00:45:09.947 Basically, as soon as question twice what. 00:45:09.947 --> 00:45:13.077 is the difference between goals at the beginning? 00:45:13.077 --> 00:45:16.757 of a logic model and outcomes at the end of the logic model ideally? 00:45:16.757 --> 00:45:18.537 and the person said, should they be the same? 00:45:19.547 --> 00:45:22.317 So let me pull up the logic model again. 00:45:23.877 --> 00:45:25.067 Am I still presenting? 00:45:26.187 --> 00:45:28.907 Yes, let me go back. 00:45:29.397 --> 00:45:31.467 Who's back? 00:45:32.397 --> 00:45:36.017 To the logic model, right? So keep. 00:45:36.017 --> 00:45:39.137 in mind, with the logic model we're looking at the. 00:45:39.137 --> 00:45:42.367 overall impact that we want to have right? And that's what's happening. 00:45:42.367 --> 00:45:45.377 at the end of the logic model. What's happening at the beginning? 00:45:45.377 --> 00:45:49.197 of the logic model or our resources and inputs? 00:45:49.197 --> 00:45:52.377 So, interestingly enough, you. 00:45:52.377 --> 00:45:54.907 don't actually the word goals. 00:45:55.707 --> 00:45:58.897 Come in your logic model right. 00:45:58.897 --> 00:46:02.097 but we do talk about goals as these. 00:46:02.097 --> 00:46:05.297 visioning items and I'm going to get. 00:46:05.297 --> 00:46:08.557 you here right to some of our examples. 00:46:08.557 --> 00:46:11.657 right? So if an overarching goal might be. 00:46:11.657 --> 00:46:14.737 improving driver and pedestrian safety on Main Street. 00:46:14.737 --> 00:46:17.757 and then the impact that we're looking at. 00:46:17.757 --> 00:46:20.917 is that there's fewer accidents and vehicular deaths. 00:46:20.917 --> 00:46:24.457 right? So remember that goal is your visioning statement. 00:46:24.457 --> 00:46:25.297 and your. 00:46:25.347 --> 00:46:27.877 Impact is the change in public condition. 00:46:28.747 --> 00:46:32.177 That you want to bring about. They should be very. 00:46:32.177 --> 00:46:33.417 tightly aligned. 00:46:34.327 --> 00:46:37.387 But they're not necessarily going to be exactly the same. 00:46:37.387 --> 00:46:39.187 and I'll pull up this example too. 00:46:39.877 --> 00:46:43.137 Right, so our goal or visioning statement. 00:46:43.137 --> 00:46:46.437 might be that we want to increase Community resilience. 00:46:46.437 --> 00:46:47.897 to violent extremism. 00:46:49.017 --> 00:46:53.007 Right, but that impact that change in public condition. 00:46:53.007 --> 00:46:53.007 00:46:54.317 --> 00:46:57.757 Right, that aligns really closely with that goal. 00:46:57.757 --> 00:47:00.797 Is that fewer individuals will join. 00:47:00.797 --> 00:47:03.917 extremist organizations, so they're not going to be like. 00:47:03.917 --> 00:47:05.297 a word for word. 00:47:06.267 --> 00:47:09.277 Lay down with each other, but they're very. 00:47:09.277 --> 00:47:12.737 tightly coupled, right, and the goal provides. 00:47:12.737 --> 00:47:15.757 the vision for your organization, the. 00:47:15.757 --> 00:47:19.097 vision, and the focus, and the impact is the change in. 00:47:19.097 --> 00:47:21.357 public condition that you want to bring about. 00:47:25.907 --> 00:47:26.457 Perfect. 00:47:26.507 --> 00:47:26.837 Yeah. 00:47:27.897 --> 00:47:28.807 See if I see any. 00:47:29.947 --> 00:47:30.807 More questions. 00:47:31.807 --> 00:47:32.557 Scream. 00:47:33.807 --> 00:47:34.467 I'll pull it back up. 00:47:34.577 --> 00:47:36.177 I need to OK. 00:47:42.407 --> 00:47:45.637 I'm in terms that the slides I can send it to all the people, everyone that's. 00:47:45.637 --> 00:47:49.057 who's attended. I have you if you entered. 00:47:49.057 --> 00:47:52.657 and put your email in, I can send it. It's not just. 00:47:52.657 --> 00:47:52.657 00:47:53.427 --> 00:47:56.767 Shoot me a email I terrorism for engine. 00:47:56.767 --> 00:47:56.767 00:47:56.977 --> 00:47:57.477 Uhm? 00:47:58.577 --> 00:48:01.697 The email that I sent you the invite but. 00:48:01.697 --> 00:48:05.047 just give me bear with me. I'll send out the those. 00:48:05.047 --> 00:48:06.517 slides shortly after. 00:48:08.097 --> 00:48:08.727 Uhm? 00:48:09.757 --> 00:48:13.447 Just look and see if we have any more questions for you, Kathleen. 00:48:13.447 --> 00:48:13.447 00:48:14.067 --> 00:48:17.187 Come here question, I hear a question for the grants. 00:48:17.187 --> 00:48:21.737 team. When will they know full expected to be released currently? 00:48:21.737 --> 00:48:24.357 you we're working on getting approved and. 00:48:24.597 --> 00:48:27.517 And we would probably think. 00:48:27.567 --> 00:48:30.897 Early spring late maybe? 00:48:30.897 --> 00:48:33.997 or maybe perhaps March we can't give. 00:48:33.997 --> 00:48:37.017 you a definite date, but we're working on it and if. 00:48:37.017 --> 00:48:40.037 you are registered with gov delivery you will. 00:48:40.037 --> 00:48:43.047 receive an email as soon as it is released. 00:48:43.047 --> 00:48:43.047 00:48:44.357 --> 00:48:47.427 Or if you come or stakeholder. 00:48:47.427 --> 00:48:50.647 that you already have. 00:48:50.647 --> 00:48:53.917 connections to our office, such as every original. 00:48:53.917 --> 00:48:55.917 prevention coordinator, they will also let. 00:48:57.387 --> 00:49:00.537 So I think a lot of people here are registered with Government Cup. 00:49:00.537 --> 00:49:02.267 delivery and if not. 00:49:02.317 --> 00:49:05.077 Uhm, I can try to put in the chat. 00:49:06.047 --> 00:49:07.657 The link to register. 00:49:16.947 --> 00:49:20.517 I have a question for you, Kathleen, just. 00:49:20.517 --> 00:49:23.587 s I see we have know based on the. 00:49:23.587 --> 00:49:27.997 cants we get and type projects they're proposing, but. 00:49:27.997 --> 00:49:31.637 a lot of people are doing trainings. Train the trainer type programs. 00:49:31.637 --> 00:49:33.987 community awareness briefings. 00:49:34.157 --> 00:49:34.607 Yep. 00:49:34.727 --> 00:49:38.147 So in terms of that specific. 00:49:38.147 --> 00:49:40.367 type of project, what advice? 00:49:40.537 --> 00:49:43.587 In a lot of a lot of the work. 00:49:43.587 --> 00:49:47.147 is dealing with people and their odds. Work is hard. 00:49:47.147 --> 00:49:48.207 to capture. 00:49:49.077 --> 00:49:52.157 A lot of the goals might be long term, but we're not going to see that. 00:49:52.157 --> 00:49:55.237 within the two years. So what is some? 00:49:55.237 --> 00:49:58.717 advice? Would you give to someone you? 00:49:58.717 --> 00:50:01.937 ean, I, we could start off more specific, maybe it's. 00:50:01.937 --> 00:50:05.137 the training type program, train the trainer, but ultimately or. 00:50:05.137 --> 00:50:08.537 whatever I could get to anyone when designing. 00:50:08.537 --> 00:50:12.017 the outcomes and the measurements, sure, so first. 00:50:12.017 --> 00:50:13.207 on people are hard. 00:50:14.817 --> 00:50:18.147 I'm I like to tell some of my colleagues that who you? 00:50:18.147 --> 00:50:21.307 t work in like lab environments, right? 00:50:21.307 --> 00:50:24.447 People behave in. 00:50:24.447 --> 00:50:27.807 their own ways in their. 00:50:27.807 --> 00:50:30.927 own self interest. So I'm like, which might be very different than what you would. 00:50:30.927 --> 00:50:33.647 be used to, and like a chemistry or biology lab. 00:50:34.797 --> 00:50:38.207 So people are always difficult and always make things a little harder. 00:50:38.207 --> 00:50:41.427 and then the other thing I would say. 00:50:41.427 --> 00:50:45.067 I stress this a little bit, but I. 00:50:47.477 --> 00:50:50.527 Uhm are well suited for pre and post. 00:50:50.527 --> 00:50:53.847 testing. So if I were. 00:50:53.847 --> 00:50:56.937 developing a training program as part of my grant. 00:50:56.937 --> 00:51:00.007 application, I would certainly look to that as. 00:51:00.007 --> 00:51:02.547 one of the ways to measure. 00:51:03.277 --> 00:51:06.647 And then the other thing I would say is yes. One of the challenges. 00:51:06.647 --> 00:51:10.107 that we face is that you. 00:51:10.107 --> 00:51:13.227 long term change in public condition that we want. 00:51:13.227 --> 00:51:16.147 to bring about right, the impact that we want to have. 00:51:16.697 --> 00:51:21.397 Uhm, we might not realize for five. 00:51:21.397 --> 00:51:24.707 10-15 years and obviously. 00:51:24.707 --> 00:51:28.047 we're not getting the these grants aren't that long. 00:51:28.047 --> 00:51:31.487 and that's why it's so important to. 00:51:31.487 --> 00:51:33.867 really, really dig. 00:51:35.097 --> 00:51:37.677 To what your outcomes are. 00:51:38.417 --> 00:51:41.747 Uhm, because those outcomes both short and. 00:51:41.747 --> 00:51:44.927 long term, or what provides you the building. 00:51:44.927 --> 00:51:48.287 blocks to get that impact. 00:51:48.287 --> 00:51:51.547 ct and they are the things that we can actually measure. 00:51:51.547 --> 00:51:54.667 right? And then we can say, my goodness, we're not. 00:51:54.667 --> 00:51:57.847 hitting the short term outcome, which means that we're not going to hit this long. 00:51:57.847 --> 00:52:00.337 term outcome, which means that we're not having. 00:52:01.077 --> 00:52:04.127 The desired long term impact that we want to. 00:52:04.127 --> 00:52:07.227 have. Let's come back and course correct, right, that's. 00:52:07.227 --> 00:52:10.367 the other thing that doing measurement and doing. 00:52:10.367 --> 00:52:13.377 it early and often allows you to do. It allows you. 00:52:13.377 --> 00:52:16.587 to course correct over time, so don't be afraid. 00:52:16.587 --> 00:52:16.587 00:52:17.377 --> 00:52:18.917 Of needing to make changes. 00:52:18.967 --> 00:52:22.237 It's great, it's much better. 00:52:22.237 --> 00:52:25.477 than going down a path that is not. 00:52:25.477 --> 00:52:26.057 fruitful. 00:52:31.957 --> 00:52:35.097 Perfect great answer I'm I don't know if I ask my john you. 00:52:35.097 --> 00:52:38.657 wanna throw a question that might help the applicants out since. 00:52:38.657 --> 00:52:40.257 we hope to have this recording. 00:52:42.957 --> 00:52:44.297 I have a question. 00:52:45.287 --> 00:52:48.977 It's along the same lines in terms of. 00:52:48.977 --> 00:52:48.977 00:52:49.867 --> 00:52:53.807 Measuring impact with people. Do you? 00:52:53.807 --> 00:52:56.907 have any suggestions on? 00:52:56.907 --> 00:52:56.907 00:52:58.077 --> 00:53:01.977 Metrics and measurements for programs where. 00:53:01.977 --> 00:53:05.017 they're engaging with the public but not necessarily in. 00:53:05.017 --> 00:53:08.457 a direct way, so you can't really do pre and post. 00:53:08.457 --> 00:53:11.597 tests. For example social media pose some. 00:53:11.597 --> 00:53:14.657 kind of public awareness campaign. 00:53:14.657 --> 00:53:17.667 where you're not coming into direct contact with people. 00:53:17.667 --> 00:53:20.917 but they're still getting the information, hopefully. 00:53:20.917 --> 00:53:25.047 right. So then you have to be a bit more creative. 00:53:25.047 --> 00:53:25.047 00:53:26.067 --> 00:53:29.087 If I can use that word and how you collect? 00:53:29.087 --> 00:53:32.127 your data, right? So you might MU if you're. 00:53:32.127 --> 00:53:35.567 able to use social media data, you might look at. 00:53:35.567 --> 00:53:38.947 click throughs. 00:53:38.947 --> 00:53:42.737 on ads. You might look at the number of times. 00:53:42.737 --> 00:53:46.507 something is retweeted or shared, right? 00:53:46.507 --> 00:53:49.547 and that might give you some sort of measure. 00:53:49.547 --> 00:53:53.747 rt of measure of you know the the larger outreach. 00:53:53.747 --> 00:53:56.267 eaching. 00:53:56.607 --> 00:53:59.747 Uhm, it. What it doesn't do, is it? 00:53:59.747 --> 00:54:03.377 oesn't get you to that point of saying, alright? 00:54:03.377 --> 00:54:06.957 someone clicked on this, they read it. That's good. 00:54:06.957 --> 00:54:10.697 but we don't then have the information on did. 00:54:10.697 --> 00:54:13.737 they learn from it? Did they change their behavior because of? 00:54:13.737 --> 00:54:16.767 it right? And that is. 00:54:16.767 --> 00:54:19.817 kind of what's missing, and that's OK sometimes you. 00:54:19.817 --> 00:54:23.007 won't be able to tell, right? But it's. 00:54:23.007 --> 00:54:26.217 being up front about what those limitations are. 00:54:26.217 --> 00:54:27.757 is also incredibly important. 00:54:31.127 --> 00:54:34.497 It just to piggyback on that. I think even if you can't. 00:54:34.497 --> 00:54:35.927 tell for the whole group. 00:54:35.977 --> 00:54:39.067 You might be able to know that. 00:54:39.067 --> 00:54:42.497 maybe do a focus group of a subset of the people. 00:54:42.497 --> 00:54:45.347 who received it, maybe to see. 00:54:46.117 --> 00:54:49.847 Did they use it? Was it helpful? I know when DHS. 00:54:49.847 --> 00:54:53.227 sometimes puts out intelligence products they. 00:54:53.227 --> 00:54:56.637 will do it. There will be a survey at the end and. 00:54:56.637 --> 00:54:59.807 you can say I this was useful to me or this. 00:54:59.807 --> 00:55:00.407 wasn't. 00:55:00.477 --> 00:55:03.687 And so there are ways sometimes to do. 00:55:03.687 --> 00:55:06.817 that. And then you just. Obviously you have a sample. We believe it went to three. 00:55:06.817 --> 00:55:10.037 ople. We've got 20 responses and just be transparent about. 00:55:10.037 --> 00:55:13.847 Maybe if you're going to do a sample survey or focus group. 00:55:13.847 --> 00:55:13.847 00:55:14.637 --> 00:55:15.667 That sort of thing. 00:55:17.587 --> 00:55:20.727 Yeah, and I think I mean it piggyback off of what David. 00:55:20.727 --> 00:55:24.107 said just a second ago too about transparency, the. 00:55:24.107 --> 00:55:27.327 one of the most important things with research. 00:55:27.327 --> 00:55:29.467 but really with measurement is. 00:55:29.887 --> 00:55:32.947 Uhm, understanding the limitations. 00:55:32.947 --> 00:55:36.447 of what you're collecting and being honest. 00:55:36.447 --> 00:55:39.497 about the information and what it says and what it. 00:55:39.497 --> 00:55:43.307 what it doesn't say in what it can say and what it can't. 00:55:43.307 --> 00:55:46.687 right? And that's what's most important. 00:55:46.687 --> 00:55:46.687 00:55:53.557 --> 00:55:56.467 Great, I don't think we have any more questions so. 00:55:57.197 --> 00:55:59.147 I guess we get thank everyone for attending. 00:56:00.277 --> 00:56:03.817 And thank you, Kathy, for the great presentation and hope this helps. 00:56:03.817 --> 00:56:04.847 all the outcomes out. 00:56:06.357 --> 00:56:09.547 Creating great implementation measurement plans and logic. 00:56:09.547 --> 00:56:10.107 models. 00:56:11.907 --> 00:56:15.507 And look up, I put in the chat a leg to the. 00:56:15.507 --> 00:56:18.717 gov delivery and our newsletter so you can keep an eye. 00:56:18.717 --> 00:56:21.677 out for the release of the notice of funding opportunity. 00:56:23.427 --> 00:56:23.807 Great. 00:56:24.677 --> 00:56:25.697 Thank you for having me. 00:56:26.737 --> 00:56:27.207 OK. 00:56:33.467 --> 00:56:37.327 00:56:37.327 --> 00:56:42.657