Presenters:
- Tran Keys
- Brianna Moorehead
- Ryan Huynh
- Ryan Lewis
Tran Keys:
Welcome. Welcome to our webinar, harnessing Logic Models for Program Improvement. My name is Tran Keys, I’m a senior research associate at WestEd, and I’m one of your presenters. I’m joined here by my colleague, Ryan Huynh, a research associate, and Ryan Huynh and I work very closely. We’re both researchers at WestEd. We do a lot of program evaluation, so we are your primary presenters for today.
We are also joined by our colleague Ryan Lewis, who is in the back kind of the, I don’t know, we call it the, behind the scenes. How about that? Behind the scenes in Zoom, and he is monitoring the Q and A throughout the webinar. So please ask your questions there. He, Ryan could easily have presented on this webinar too, so know that all the responses he’s flagging for us as well. All the questions, excuse me. And as much as he can given the the frequency of questions, he will also respond. Finally, we have our team members, Brianna and Colleen on the call, and you’ll see their name with the tech support in front of it.
All right, so for those of you who are, who don’t know WestEd, WestEd is a nonpartisan research, development, and service agency, and we work to promote excellence, improve learning, and increase opportunity for children, youth, and adults. Doesn’t say on here, but our vision is basically thriving learners, empowered communities. That’s the work, our through line in the work we do is very much that.
Okay, so our agenda I mentioned is quite meaty. The topic is logic models, and you know, full disclosure here, Ryan Huynh, what Ryan Lewis and myself, we run logic model workshops and this really could be a workshop, but we know that’s, you know, that’s too much of folks time. So today is what I’ve heard Ryan Lewis call an appetizer. We’re gonna give you an appetizer on logic models, okay? And so the agenda here is a little bit of a welcome and getting to know you through polls. We’re gonna touch that next and provide a logic model overview. We’re gonna show you two examples. After we give you the overview, we’d like to walk you through, briefly walk you through two examples of logic models of course, and talk about takeaways, right? We’re calling it harnessing logic models for program improvement, so we wanna end with some takeaways.
We’ll also touch on Q and A. The questions that come in the Q and A Ryan Lewis will lead that section, and then we’ll close out and share resources with you in the end. Okay, so here’s the welcoming and engagement piece. We’d love to know, this is a really large group here that’s joined us. So the best thing we could do is think of this polls. You wanna get us to know the group here? What type of organization do you represent? Brianna just launched a poll here. Let us know where you’re from. And let’s let folks see the results here. We have a good mixture. We have predominantly folks, or the majority of folks are from nonprofit organization or higher education institutes of higher education. We have folks from private sector consulting firm.
Good to see our SEAs here in county offices of education school districts. That’s really a big part of the work we do are with our education colleagues in the local, regional, and state level. Always good to see. And it looks like we even have someone from a school site, which is always fun for me to see folks from schools. So thank you. Thank you for participating in our poll for us to get to know who’s in the room, where you’re from. The next poll we wanna ask is, what is your role?
Brianna Moorehead:
Here are the results, Tran.
Tran Keys:
Oh, look at this. So we have, oh my goodness, we have colleagues, Ryan Huynh, Ryan Lewis, that are evaluators and researchers. We have program administrators. Happy to see program staff, district staff as well. We will definitely be speaking to you in the work, the great work that you do in the field. So we’re definitely gonna tap into that. And we have a pretty big section of other, which I’m curious what they are. And I know not everyone can see what the responses are because of the way we’re setting up the webinar, but I would love to see it. So feel free for those of you who said other to tell us what it is, and Ryan and myself will at least be able to see where, what other means for you. Okay, Brianna, our last poll, our last poll is really for Ryan and myself to get a sense of your familiarity with logic models, okay?
So on a scale of one to five, where one is not at all familiar and five is extremely familiar, we want to know how familiar are you with logic models? And of course you know, I’m gonna say this, there’s no wrong answer. The majority of folks have responded to three. So in that very middle familiarity, we do have folks who are not at all familiar. And of course we’re not surprised, right Ryan? Because we saw in the previous poll, we had colleagues who are researchers and evaluators. So of course they would be familiar with logic models. So we also have folks who marked four and five for their familiarity with logic models, okay? So we’re gonna ask you to do a reflection before we cover content here. So if you’re a pen and paper person, go ahead and grab your pen and paper. If you are note taking on your laptop, on a e-document, please make your way there.
We want you to answer this question. We want you to reflect on a current or upcoming program that you’re working on, okay? Think of something that you are currently working on or something that’s coming up. What is the program’s primary goal or purpose? Who are the key groups or individuals involved in the program? And what outcomes do you hope to achieve? So think through those questions and jot down your responses. We actually want you to make this as an exercise for yourself. Keep these questions in mind as we explore how logic models can enhance program planning and program implementation. Alright, so our objectives for the webinar is we hope that you will leave us with understanding the purpose and components of logic models. We wanna learn how to develop them and refine them for program planning, program improvement and program evaluation.
And finally, we want you to understand a little bit more about how to apply logic models to enhance your communication, to track outcomes and to guide your decisions. So before we get into the components, I wanna share with you a little bit on the, an overview, a high level overview of logic models, and then Ryan will take you to through the components. Those of you who are very familiar with logic models might have seen this cartoon. It makes its way in our evaluation, research and evaluation circles, right? When we need a little chuckle in our work. And so what you see here on slide 12 is a slide that says Simplified Logic Model. And you’ll see a cartoon that humorously illustrates the, what we would say, an oversimplification of program planning.
So it depicts a model where funding, which is the give us money, is where the, so funding, which is represented here as give us money is expected to directly lead to success, which is we all win, right? So give us money and we all win, but there’s no explanation about the process in between. So for us, this highlights the need for a structured approach that establishes a clear through line between resources, activities, outputs, and outcomes. So we hope that you leave today’s webinar with a deeper understanding of how the components of the logic model work together to drive your program’s impact. So who knew a simple cartoon could be so, has so much meat in it, right? So I really love this cartoon to share out before we talk about logic model development.
So what is it? What is a logic model? It is a graphical representation of the relationships between the parts of a program and its expected outcomes. We often refer to logic models as a framework. A framework for program planning, program implementation, and program evaluation. And then finally, an another definition that I really like, it’s a longer one, but it really wraps it all up. It’s a systematic and visual way to present and share your understanding of the relationships among the resources that you have to operate your program, the activities you plan, and the changes or results you hope to achieve. So that long bullet point at the end that should look familiar to you from what we asked you to do in terms of reflecting, right? What are your goals of the program, your purpose, what key groups, and what outcomes, right? So those all are wrapped up there.
I won’t go through the graphic here that talks through what each of the components are because my colleague Ryan Huynh will cover that. But we want you to see basically that resources and activities are what you planned, your planned work, right? Resources and activities. And everything else beyond that is what your intended results are. And again, Ryan will go through this in greater detail, but this is basically the flow of the logic model. All right, another good, very good question is why, why even bother creating a logic model? So a lot of people say, you know, who aren’t engaged in this work and think, gosh, it’s just another thing to do and I have it here, Tran, I know what my program is in my head. Well, my response is, well, people aren’t in your head. People can’t read your minds. And so, and you work in teams and then you have clients and you have partners, right? And you serve communities.
And so it really is wonderful to have what you have in here that you might know out on paper, right? Because it establishes a shared language and a vision for change. It clarifies connections between your program’s components and what you intend to accomplish, your intended impacts. It really guides program evaluation and decision making. And you’ll hear us say this often because when I introduce us, I mention that we are program evaluators as part of our work at WestEd. And when we are engaged in an initial call with a partner who wants us to conduct a program evaluation, guess what our first question is, may we see your logic model? And if they don’t have one, guess what? We write it into our scope of work that we will be helping them co-develop a logic model.
That’s how important we feel logic models are to being able to guide program evaluation and decision making. And then finally, it’s the communication component, effectively communicating your program goals and operations to your funder, to your staff, to your team, to the community. So there’s lots of good reasons why we suggest you create a logic model. All right. Ryan, I’m ready for you to go ahead and take over and walk us through the different components of the logic model.
Ryan Huynh:
Great, thank you, Tran. Thank you everyone for attending. So when it comes to thinking about what makes up an effective logic model, there are sort of two criteria that come to mind. The first is that your logic model follows a core structure. And the second is that your logic model is adaptable based on the changing needs of your program or of your environment. So if you can strike a balance between that structure and that adaptability, then you’ve got a solid foundation for your program planning and implementation. So while you may might make adjustments to your logic model based on the unique needs of your program, I would encourage you to stay consistent with the core structure. And that’s what I’m gonna be diving into. We’re gonna go through the key components of a logic model to help us understand how a program works and what impact the program is aiming for.
So this is gonna include resources, activities, outputs, short-term outcomes, medium-term outcomes, and long-term outcomes. I’m gonna dive into each of these components one by one. And as I do, I want folks to consider how all these pieces fit together to make sure that we’re intentional about our program design and we can track whether our program is making a real impact. All right, so at the core of any strong logic model is the problem statement. This is where we clearly define the issue that we’re trying to address. So when we have a well-crafted problem statement, it helps ensure that our activities and our outcomes are aligned with a real identified need. So in crafting your problem statement, I wanna share some questions in the chat that would be helpful to reflect on and answer as you come up with a problem statement for your program.
First question is, what’s the specific problem that the program is trying to address? The second question is, who is affected by it? And this last question is important because sometimes we find in our work that folks need to spend more time reflecting on this before diving into the actual program planning and implementation. But the question is, what evidence do we have that this is a real issue? So when we ground our logic model in a clear data-driven problem statement, we set ourselves up for success. Having it clear and data-driven helps keep the focus on the challenges at hand and ensures that everything we implement is purposeful and strategic. All right, so once we’ve crafted our problem statement, we can start to think about what’s required to make this program work. And that’s where the step of mapping out our resources comes in.
So resources, also referred to as inputs, these are all the essential elements that support program implementation. So we’ve listed here some examples. Resources could include anything such as human resources, so all the people, staff, consultants, volunteers that contribute to the program. It can include monetary resources, so think of all your funding streams such as grants and institutional support. It could also include physical resources, so think of what facilities we have access to, the technology, the equipment, and also educational resources such as curriculum, training materials, and also research tools. I already saw a quick question in the chat. Can community voice be considered resources? Definitely community input can also be a resource. And of course we’ve got time, often the most limited but most valuable resource of all, time is a resource that is to be considered when mapping out your logic model.
My point here is that before we can take any action with our program, we need to be realistic about the resources that we have to work with, and also consider the resources that we might still need to secure to implement or improve our program. So we’ve mapped out the resources we have, and we move on to mapping out our activities. So these are the specific actions we take to achieve our program goals. We’ve listed here some examples. So program activities could include running professional development sessions. It could include providing family support programs, implementing new policies or procedures, using curriculum or teaching practices. It could also include offering mentoring or coaching, and developing new learning materials.
Ultimately, activities are what drive our program forward by taking the resources that we have available and turning those resources into meaningful action. All right, so we talked about what goes into a program, your resources and your activities. The next question is how do we measure what comes out of what we put in? And that’s where outputs come in. So outputs, these are the direct measurable products of program activities. A good way to remind ourselves about outputs is emphasizing the word measurable. Really emphasize measurable, because outputs usually serve as evidence of program implementation. So think for example, maybe our program is tied to requirements from a funder.
Our program is being provided funding, and we have certain requirements that we need to measure as evidence of our program implementation. So we might measure the number of activities conducted by our program, the number of participants of our program, or even the number of materials developed by our program. We might also measure what new policies or procedures were put in place by our program. Long story short, outputs tell us that our program is up and running. And one key distinction that I wanna make between outputs is that they don’t necessarily tell us whether the program is making an impact. So that’s the distinction I wanna make as I dive into making our outcomes explicit. You may have already noticed when I shared that overview earlier, that we split our outcomes into three separate components based on short-term outcomes, medium term outcomes, and long-term outcomes.
And when Tran and I do our work with partners, we recommend every logic model to do so for multiple reasons. The first is that dividing into these three separate components provides a more clear path for measuring impact to our program. And it improves measurability and accountability, and also supports program adjustments, especially as we consider how to evaluate our program. So diving into that progression of impact, we start with these short-term outcomes, which are usually the first signs of change that happen as a result of our program activities. We can usually see these short-term outcomes fairly quickly after implementation. So some examples might include a professional development session that we’ve held might lead to teachers gaining new instructional strategies.
Another example might be we’ve conducted a student reading program and it’s led to improved phonic skills for students. Another example could be running a community health campaign and it’s lead to greater awareness about nutrition. So overall, these kind of changes in knowledge or skills show that people are learning and engaging with the program. And that’s a great beginning step toward long-term impact. So as our program continues, the short-term changes start to influence medium term outcomes. So think changes in behavior, attitudes, and practices. Usually these take a little longer to develop because they require participants to not only learn something new, but also start applying it consistently.
So for example, a teacher might start integrating new instructional strategies into their classroom, or parents might start becoming more actively involved in their child’s education, or another example of a medium term outcome might be students developing stronger reading habits over time. When we start seeing these kind of medium term outcomes and these shifts happening, we know that the program is having a deeper impact. Primary at the highest level, we have long-term outcomes. And you know, this is the ultimate impact that we’re striving for with our program. Long-term outcomes might take years to fully materialize, and they might even depend on external factors beyond just the program itself.
So some examples of long-term outcomes might include higher student achievement rates, improved graduation rates, policy changes that might lead to lasting improvements, or even stronger community engagement in education. Since long-term outcomes are often hard to measure in the short run, it’s important to have a system in place for tracking progress over time. So that’s where that evaluation conversation comes in. All right, so to tie everything together, we’ve gone through each of the components one by one. We have this annotated logic model here on this slide that shows how all these components connect to create meaningful change. So I’m just gonna reiterate one more time so that this flow in the logic model is ingrained in our program planning and implementation.
We start with resources. These are the inputs that allow us to run our program. These resources support our activities, which are the actions we take to address the problem. And activities lead to outputs, which give us measurable evidence that the program is running as intended. And in the short-term, we start to see knowledge and skill gains. And over time we start to see changes in behaviors and practices through our medium-term outcomes. And last but not least, ultimately, if the program is sustained and effective, it contributes to long-term systemic change. And last but not least, I also wanna call attention to this section at the bottom for additional considerations. As we complete our logic model, it’s important to take time to consider any other factors that are pertinent to our program that may not fit into these other logic model components.
So some additional considerations that I’ve seen be implemented into post logic models is maybe external factors such as a policy, as an example. With that, the big idea is that this logic model serves as a blueprint to ensure that our program is one, intentional, two, measurable, and three, impactful. So we’ve talked about the what and the why behind creating a logic model. The next logical question you might is who, when, and where. I’m gonna pass it back to Tran to break that all down for us.
Tran Keys:
Thank you Ryan. And before Ryan, before you move forward, I just wanna say I loved, I love this slide, I love seeing everything. I appreciate you walking through each component, and I love seeing everything in one slide. And it’s reminding me something I forgot to say earlier, which is we recommend when you do develop a logic model that it ends up being a one page, just like this annotated logic model. And the reason is it’s just a power to be able to seeing everything in one view. And some people say, ah, that’s really small font, Tran. That’s like really difficult. Well, I said one page, but I didn’t say what size it needed to be. And I say that because guess what? The folks we work with to help build logic models, after they finalize the logic model, they blow it up at poster size. And so then they put it in spaces that remind them of their work, right? Because a logic model really is the story of your program. And so seeing it in one big slide or one big poster, slide that you make into a poster is quite powerful. So thank you Ryan for walking us through all that and showing it all together in one slide.
As Ryan said, there are other considerations, right? In terms of many of you are here because you wanna build a logic model. So we wanna give you some tips from, based on our experience working in this area quite a bit. So who should be involved in developing a logic model? You want representatives from key groups, and here’s a list of folks, people who would play this role. So program leadership would be important to have their view, a staff responsible for implementation. That is key. The people who are running, who are doing the program need to be part of building the logic model. This is very, actually quite rare, but so powerful when you can have a participant beneficiary of your program, right, to be a part of the conversation of the logic model. And then funders and decision makers, and then evaluators or researchers.
Oftentimes, especially I mentioned earlier on, Ryan, Ryan, and myself are program evaluators. And the first question we ask potential clients and partners is, do you have a logic model? And if they don’t, we say, well, let’s build one together. That’s how critical it is. And so if you are in a, if your program has an evaluation component, you might want to include that person in it. Next is when should a logic model be developed? So let’s talk ideal first. Ideal time to develop a logic model is during program design as part of your program planning. And then after that, you would modify and enhance the logic model as the program evolves. A question we often get asked is, is a logic model a one and done activity? Absolutely not, because your program will change over time, and you want that reflected. So a nice little tip we give is when you develop a logic model and you quote unquote finalize it, date it at the bottom right, left hand somewhere corner, right? Where you say, what is it? February, February, 2025, right? Because maybe in six months there’s big changes to your program, you’re gonna wanna revisit your logic model and update it.
And it’s really nice to actually see progression of logic models over time. So that’s the ideal time, is during program design. But the reality is really any time after your program has begun, and I really do mean this because what did I say when Ryan, Ryan, and I asked potential partners about being evaluators, and if they don’t have a logic model and their program has been going on for a few months, that’s okay. We’re gonna help develop one now. So don’t let that stop you from developing it, you know, if you hadn’t had one, right? It’s never too late, is what we say with logic models. Next is where? It sounds like a weird question, but in this time it actually is very relevant. Where should we, where should a team develop a logic model? If you are all together because you work in the same building, right, it’s that kind of environment, in person, all the way. That’s perfect. And we’re gonna show you an image of what that could look like.
This cartoon image was nice to see, you know, a bunch of stickies, right? Whiteboard, things like that. So in person, bringing folks together, making sure people understand why they were invited, that’s really important too. And then now really we do so much logic model development in the virtual space, and why would we even do that, right? Well, meeting our partners and our clients’ needs. So sometimes we work with a group, a team that wants to have a logic model. However, they are not in one location in terms of everyone who should be at the table, the logic model development table. And so we do this via Zoom, and again, it works out perfectly because of the way, right, it’s really called virtual facilitation. So taking all the best practices we do in person and translating it to make sure it works in a virtual space. And we are very confident about this, and that can work out really well.
The third one is doable. It is the most difficult, the hybrid. The hybrid means you have some people in person together in a room, because they happen to be working in that same space. And then you have some of their colleagues who really should be a part of this too, who are not in the space. So they have to zoom or virtual, right, come into the meeting virtually, it can work. It’s a little bit more complicated, however it can work. And the little tip there is to make sure you have two facilitators, because one of the facilitators is the person who is tied to the laptop communicating with the virtual folks to make sure nothing is being missed, right? Because things will be missed if you have that people in person and people virtually. That’s just the nature of our communication.
So to put extra safeguards to make sure all voices are heard in the development of the logic model. So we wanna show you, I think it’s a photo next of what, it’s cartoon photos of next of what it is we suggest. This sounds a little bit nitpicky, right? But we, you see Sharpies there. We do mean Sharpies. We don’t, we say this because again, we’ve done this very frequently, and now we always use Sharpies because if you can write it, folks write it clearly, right? You can see it from a distance better instead of everyone using just the regular pens that they write. It really makes a big difference in terms of people seeing what is written. So I’m kind of being very specific here, but I again say Sharpies not regular pens to write on the post-its. And even with the post-its, oh my goodness, I’m gonna be nitpicky there. Make them light color, right? We can buy packets of post-its, and some of them are really dark colors.
And so one time we did a workshop, and they were pretty dark colors and they were hard to see. So you see our images here are light colored and multicolor as well. Stick the stickies because each of the colors you assign to a component of the logic model, right? So resources are yellow, activities are orange, et cetera, doesn’t really matter the color. And then of course the wonderful writing charts, right? That you can have stick, stickies behind them. And so we wanted to give you very concrete advice about how to do that in person. And then you’ll see here in our next slide, this is an actual session we did, I think, I wanna say two years ago, Ryan, at a K, 12 school district. And this is what it looked like.
The top, are the chart papers with resources, each component of the logic model, resources all the way to the outcomes, separated as Ryan Huynh said, short-term meet mid-term and long-term. We also had additional considerations. We actually had the problem statement too, but it didn’t make it to this page, but it’s empty as you can see. And our direction for that, this is was a four hour session with a team of about, I think it was about 18 folks and the leadership of an expanded learning program. And at the end of four hours, this is what they generated. And this was just the first iteration, right? And so we want you to get a sense for what it looks like, and it was so satisfying for us and for them, more importantly for them, to see them map out what their program looks like in terms of what they have and then what they want to accomplish.
Okay, so that’s what the in-person looks like. And then we wanted to show you, we said virtual can work out really well too. And what we have found that works really well, and I know that there are other tools out there for sure, the one that we used and really find effective is Padlet. Okay, so if you are not familiar with Padlet, take a look at that. It just is a wonderful tool to use in a virtual space to build out your logic model, the same idea, listing all the components of the logic model. And then you get folks to, you know, add on their thoughts on each of the, what should be under each of the component. Okay? What we’ll say here is when you develop a logic model, you wanna make sure every team member’s work is included. This is why in that list of who should be in the room was not one or two people.
It should never be one or two people building a logic model. I’m gonna just put that out there. I know sometimes with time constraints and so much in our plates, we feel like we should be the, we are the only one who can do it. We, the best logic models are the ones that are co-created with the right people in the room. Okay? All right, and so in the next slide we want to share with you two examples of logic models that we’re not going in any great detail. What I really wanna do, for example, in the first logic model is to show you, you heard Ryan talk about the different components and how it’s laid out, but this is exactly what Ryan Huynh walked us through in terms of what a logic model covers, what it should look like, for me, a visually appealing format, very clean, very clear.
And so this one, again, I’m not gonna go through the content of it, but just so you know, AMMP is a program that stands for Afterschool Middle Grades Math Program. And so this is listed in one of our resources for you to look at as an example of how a logic model is being developed, okay? And again, all the components that Ryan Huynh walked us through are in here. And it’s one page, Ryan! I’m so happy to see it as one page. This looks so good. And if it’s blown up, you can imagine, right? It’s a being blown up, being very clear. The second example I wanna show is, I like this sample logic model because it shows components and features of the program. And this program specifically is a science teacher professional development program, okay? Science teacher professional development program. And I like this one because it shows components and features of the logic model, sorry, of the program and the associated evaluation questions at the bottom.
And again, I opened up our webinar by sharing with you that the three of us, Ryan, Ryan, and myself, are program evaluators, and this group put their evaluation questions in the same document as a logic model, which I find very powerful, right? It helps to see alignment is why I find it powerful. The other thing I will say about this logic model is it takes more time in thinking and mapping out, but I also love the arrows that show directly what of the components lead to, which of the next components, right? And so some of them will touch multiple things as you can see. And it’s just a nice way of showing the thinking behind why the, why this group put together what they did. And again, a very visual, simple and visual logic model. So in this section we wanna really briefly talk about some take takeaways from why we feel logic models are so important for program improvement.
And so this slide here outlines really various ways to harness logic models for program improvement, really by using them as a framework to align your activities, your resources, and goals. So you see here we talked already about enhancing communication, which is fostering a shared understanding of goals, processes, and alignment across all involved individuals and groups. Guiding design and implementation, right? Which is ensuring program components are purposefully aligned with intended outcomes. And I heard Ryan Huynh used that word as well in describing logic model development. Always to drive continuous improvement using data and insights to refine the strategies and strengthen your alignment over time. And finally, the bullet point of strengthen your evaluation, focusing measurement efforts and tracking how the inputs and the outputs and the outcomes align.
And I have a little note here about set measurable metrics, right? Defining clear indicators to assess alignment, progress, and success. If metrics are of interest to you, we do, in the next slide, you’ll see just a screenshot really of, you’ll see a screenshot of a publication we put together a couple years ago. It’s called Aligning Data and Measures to Outputs and Outcomes of a Logic Model. It’s a really short document that does just that, align data and measures to the logic model. So when we were putting together this webinar, we really wanted to include this topic, but then we realized we would have to ask you to attend a two hour webinar, which is too difficult. All right, so I’m going to now pause, take a breath, and I’m gonna invite our colleague Ryan Lewis. I’m asking to come off mute to share some of the questions and possible responses, Ryan, with us.
Ryan Lewis:
Yeah, hello everyone. Great to be here. We’ve had some fantastic questions in the chat, so I’ll definitely steer folks to look through those questions that we’ve already answered. But we had some really good questions there about what was the difference between the theory of a change or theory of change in the logic model. And so we gave our definitions there. And then also a fantastic question about thinking about how you keep folks engaged through the process. So I talked about a little bit about, you know, creating manageable chunks of work, so not trying to tackle too much at one time, and also doing maybe shorter sessions but on successive days so folks can come back fresh the next day. But you can also kind of stop at the end of the day and say, okay, this is good enough for now.
And it really reminded me of another question that we get a lot, which is, you know, can your logic model be changed or modified over time? And the answer is absolutely yes, right? Like we stress all the time that this is a dynamic living document, and it really should evolve as you come up with more ideas, as you understand new thinkings about your programs or the context in which they lie. And that’s exactly why Tran instructed you to put a date and a version number on your logic model, so you always have a timestamp of what was your, your thinking at the time. But also we know that these things will evolve and, you know, things about the way that you deliver your programs will probably evolve over time as well. So great questions there. We have a new one that came in, and I’ll keep answering these as we go, but a great question from Maria. Hi Maria, it’s great to see you.
How narrow, in scope should the logic model be? Whether you should focus on one effort versus more than one effort? It really can depend, Maria, I would say, you know, every program can be different, and it’s really, I would tend to think about it as, are you thinking about an independent program that operates on itself and then you have other programs that act as complimentary or maybe like, you know, sibling programs versus are you really trying to describe a program that is complex and has multiple phases? So in the first instance, I would say that multiple programs that act as like siblings or complimentary to each other should be their own logic models. It’ll just be a little bit easier to think about them clearly in that way.
But if you really do have a complex program that just has multiple phases, it might be easier to just split them out and either do a logic model per phase or do one very large logic model that outlines clearly what the phases are. So it really can depend, but I would just think about kind of the structure of your program. But yeah, anything you wanna add there, Tran and Ryan, to any of those questions that we answer?
Tran Keys:
Yeah, the last one from Maria, Ryan, and yes, hi Maria. Thank you so much for joining us. The example I can give, Ryan, completely concur with everything you’ve said, and will add that there was a group we worked with, that I worked with, with another colleague, and it was a really, a really big Head Start program. Head Start, Ryan. And those of, you know, Head Start, it’s a very big program, and it was that highest level of leadership who wanted a logic model developed, Ryan. And what happened was, we, our talk was to them was you gotta make sure the right people are in the room. So everyone who was the lead of the smaller components of the Head Start program were invited to help build the larger logic model for the organization.
And guess what, when we were working on it, you could hear them say, wait, well that’s too nuanced, that’s too detailed. And then someone else said, well, maybe you should have your own logic model. Ding, ding, ding, ding, right? So it made them, so they helped build the organization’s big logic model, and they found it so useful that their smaller team, right, did the same thing, took the all the workshop materials and then did their own logic model within their smaller, you know, team of, I wanna say 30, right? So the bigger organization, right, went into a smaller group and did their own logic model because they had different inputs and activities and outcomes and outputs. But the fact that they built a larger one as the bigger organization of leadership, it helped informed and aligned the smaller logic, not smaller, but the smaller team’s logic models. So that’s the thing I would add. Wonderful question there.
Ryan Lewis:
Yeah, I think, you know, we often coach folks, we’re starting, you know, this can, it can feel a little bit like a rabbit hole, right? Where you start getting into the details of your work and the scope really feels like it grows and grows and grows of what you’re trying to describe. So, you know, definitely keep it simple when you first start, just try to really, you know, describe the core components of what you’re doing, and then naturally, as you think about what’s essential, start going in more and more details. But we try to, you know, try to have it be, you know, really true to a one pager, very simple, you know, clear description of what you’re doing in the beginning. And we just got another question in chat, which would be great to cover, which is, can you elaborate on the distinction between outputs and outcomes, which is a great question.
We get this question all the time, and I’m gonna actually post the, here, I can post the definitions that we use for both in the chat. I think it’s helpful to just see those because sometimes just, you know, kind of reading the definition side by side can be really helpful. But this really is how we think about the distinctions. Outputs are the direct to measurable products, the program activities. So think of them as very countable things, right? If you’re delivering a workshop and you had 40 attendees, you know, that’s a countable output of your work is that you had a workshop and 40 people participated. Outcomes tend to refer to things that are, that change. So those can be countable, but the, you know, they also are thinking about more, you know, what happens afterwards. So because those 40 people attended, you know, what was changed for them, what’s different for them now?
That’s really kind of where you get into your distinction of what an output is, which is just the direct result of what activity you’re doing. And it can be very, very plain and simple and very countable versus what actually happens, what’s the next domino that happens because of that output. And I’ll pass it back to Tran and Ryan to take that one too, because we get this question a lot.
Tran Keys:
This is a great question, and I’m glad someone asked for a clarification, too, Ryan, because like you said, we always get folks, and quite frankly when we facilitate workshops, we will tell them, we will repeat what it is, and we will have to move our stickies. So that’s another thing, that sticky activity you do sometimes people put things in the wrong area, and Ryan, Ryan, and I feel a responsibility, right, to make sure everything is what it should be in terms of what they, what the program components are. We will move some of the things people think are outcomes into outputs. And Ryan Huynh described it really, really well for me in terms of outputs are the, it’s your evidence of implementation, right? So think about it. Let’s say you have funding for a program, and they ask you for all of this data to collect that, as Ryan Lewis mentioned, are very tangible, right? You collect numbers of attendees’ sign in sheets, right? Things are about monitoring, right?
Result, the data you collect from monitoring your program, that would fall solely under outputs. Outcomes are what you wanna aim for, right? And again, Ryan Huynh, love the breakdown with short-term, medium term, and long-term. Because short-term is about changes in knowledge, which you can capture pretty quickly, right? You give a teacher, a teacher some professional learning in an area and you ask them for their change, or you test their change and knowledge. And so that’s a short-term. Now if you ask them to change a teaching practice, that becomes medium term outcome because like Ryan Huynh mentioned earlier, it takes a little bit of time to change behavior. And then you get to the long-term, which is really what we want.
We wanna have systemic changes, right? In our student outcomes and teacher outcomes, whatever outcome it is that you’re working on. So really fine distinction, thank you for that reminder to remind folks the differences between outputs and outcomes. Wonderful, well thank you, Ryan, so much. Those were wonderful questions indeed. So we’re almost wrapping up here. And so we, earlier on in our webinar, we had you do some reflecting of your program. And now what we wanna do in the next slide, and we’re also put it in the chat is Brianna’s actually gonna open the chat up for us. And we want you to apply what you’ve learned, and we want you to share with us in chat just to pick one of these questions. What’s one key takeaway from today’s session? Or respond to the second one, how might you use logic models in your work, even if it’s just a small way? Thank you so much, Ann, for starting us off.
Logic models help establish a shared vision, absolutely. Absolutely. Especially right, Ann, when you have the right people in the room helping to develop, co-develop the logic model. Clear guidance on what info, what information goes under each element. Thank you, Val, so much. Clear guidance on what info goes under each element. I really like the suggestions about keeping developers engaged and energized. I plan to use that. Yes, and then candy, snacks for a four hour session really helps, breaks, right? Make sure you got people stretching and in fact, the activity we always do makes people walk around, right? With post-its and writing and getting up. And so yes, make it very active. Make sure the right folks are in the room, not just a handful of senior leaders. That is absolutely a great takeaway.
And again, from experience when we have worked with senior leaders and we help them build the logic model, what we end up having to do is have a list of questions we have to ask other people, right? Because senior leaders might not know all the details that would be helpful to develop a logic model. No dark post-it notes. Yes, you try that and tell me if I’m, was exaggerating, Eric. Here’s our contact information. And so really, you know, we’d love to continue the conversation with you. We have resources that when you get to slide deck, you’ll be able to click on the links and access the resources. So make sure you do that. And then finally, WestEd, visit our website. We got lots of good stuff on here, lots of really good resources from, you know, you think of the education, you know, research program evaluation topic and you’ll find some support.
And we’ve recently revamped the website, and it just, we’ve gotten such great feedback from the field. Please go ahead and visit that website, and you can also sign up to stay connected with us. Many of you are already connected, which is how you heard about the webinar. Ryan, Ryan, and I, and Brianna and Colleen would just like to say thank you so much for spending this hour with us, and we hope to see you on another webinar.