Leigh Parise: Policymakers talk about solutions, but which ones really work? Welcome to Evidence First, a podcast from MDRC that explores the best evidence available on what works to improve the lives of people experiencing poverty. I’m your host, Leigh Parise.            

This is one of a three-part series on pre-K student assessments, and, in this episode, we will focus on the perspectives of center directors in the assessment process.

The majority of children in the United States now attend some type of formal pre-K program before starting elementary school. Pre-K assessments—or short tests and activities that measure early skills—are an important tool for understanding children’s learning and development in these settings.

However, pre-k programs are unique from K-12 schools because they typically decide on their own which assessments to use, and there's no systematic approach across pre-k programs for understanding what children know.

Pre-K center directors play a critical role in deciding how to collect assessments, helping     teachers understand the data from assessments, and using this information to make important decisions to strengthen teaching and learning. To truly improve early learning and assessment systems, it is important to understand the perspectives of pre-K center directors to ensure that the needs of schools are reflected in how measurement tools are designed.

On this episode, we talk with two leaders from the AppleTree Institute for Education Innovation. AppleTree is a nonprofit organization consisting of a research institute and a network of exemplary charter preschools in Washington, DC, working at the intersection of research, policy, and practice.

Dr. Niesha Keemer is the Principal and instructional leader at AppleTree’s Parklands campus, where she has worked with three- to five-year-olds for the past six years. Dr. Abby Carlson is the Director of Research and Impact for the AppleTree Institute, where she works on developing and implementing the organization’s instructional model across a diverse set of schools. Today, I talk with them about how their schools conduct and use early learning assessments and hear their perspectives on how assessments could be improved to meet the needs of pre-K centers.

Abby and Niesha, thank you for joining me.

Niesha Keemer: Thank you for having us.

Abby Carlson: Thank you for having us.

Leigh: Niesha, I'd love to start with you. Could you tell us a little bit about AppleTree schools and what role you play at your school?

Niesha: AppleTree is an early learning public charter school network work in Washington, DC. Here at AppleTree, we provide free, high-quality preschool to three- and four-year-olds. And at AppleTree, we believe that we could be instrumental in closing the achievement gap before children enter kindergarten. So as a principal or instructional leader here at our Parklands Campus, I support teachers so that our students receive engaging learning experiences that actually support their overall growth and development.

Leigh: Abby, I am excited to get to hear about your work. Could you tell us a little bit more about the role of AppleTree Institute and your job as the Director of Research and Impact?

Abby: Sure. And you said a little bit of this, but the institute's really focusing on that development, refinement, implementation of Every Child Ready. And that's our instructional model. Niesha and a lot of other school leaders are implementing the model in their schools. And as part of that relationship, we get ongoing feedback. And that goes back into our model refinement.

My role is to kind of guide that research agenda. I work across the implementation and the model development teams to figure out what's working, what's challenging, and kind of what needs to be true under various implementation conditions.

Leigh: All right. Abby, I think this next question could be for you. Abby, in your eyes from both a school and an organizational level, what role do the classroom and child assessments serve at AppleTree?

Abby: it's a good question. And I would say talking about the role of assessments as, especially in early childhood, I think is always really challenging because they serve so many purposes. And I think we often think about the fact that we need to measure student learning at the accountability level. We're accountable to the Office of the State Superintendent of Education, to the D.C. Public Charter School Board. It varies by state, but there's always some sort of accountability framework or metric.

We're also accountable to families. If you're entrusting us with your child's learning, we need to show you what that learning looks like. And to do that, we need to measure it. So, I think those things are kind of top of mind, especially at the instructional quality team of the AppleTree Institute level. But I don't think most people get into early childhood because they love accountability frameworks. I think we're mostly here to provide teachers with the best possible supports to help children.

All this is to say, I think the role of assessments is really to guide instruction. And this is actually a decision that we really had to face head on when COVID started. We had to sort of pare everything down in March, 2020, and really the following school year as well and say: okay, if we can only do so much, if we can only assess so much, if we have very, very limited time with children, what is the purpose of measurement?

In years past, we always had a lot a back and forth where I'm like: okay, well, the purpose is for instruction, but the purpose is also for accountability and we have to report it to other people, and on and on and on. But it was actually an opportunity for everyone to say: hold on a second, if we really have to make a choice if this is a forced choice, it's guiding instruction.

And we made a lot of changes based on that, which has actually been, Niesha might not agree, but I think very exciting over the past year or two. We've really started to shift a lot of what we're doing to put much more emphasis on that idea of guiding instruction. So, all measurement, all assessment should produce actionable reporting that teachers can use to make decisions. And that's kind of been our really strong approach.

Niesha: In the classroom, the teachers meet with the students in individual flexible small groups. And then they meet and they teach them on that at the given skill. And that was determined by the baseline assessments. And after receiving a few lessons and direct instruction on that skill, the teachers administer the “check for understanding.” And if the student successfully completes that check for understanding, this means they've mastered the skill and they're now ready to move on to the next skill.

So, teachers keep track of their progress, the student progress by entering data into the Every Child Ready assessment application. And this application supports custom small groupings of our students, and it directly links back to the Every Child Ready lessons. Everything is connected. Teachers are following the lesson plans, Every Child Ready lesson plans, they're implementing lessons. They're then doing checks for understanding and putting the data back into the app.

Leigh: Great. Thank you. All right. So you've both now mentioned Every Child Ready. Can one of you share a little bit more about what that is specifically?

Abby: People often say it's a curriculum, but I would say it's a whole instructional model. It's a curriculum, professional development wraparounds and adult learning, and then the assessment piece as well. So, sort of like how to measure success, what to teach, and then how to teach it.

And AppleTree schools, Niesha’s school, that's like the primary partner that we work with to do ongoing refinement for the model. But there are other schools that use it in DC, and we have some New York and Texas partners as well.

Leigh: I would love to hear a little bit about how you think about teacher's relationship with the assessments. My last year of teaching in New York City was the first year of third-grade assessments at the state level. And it was a lot as a teacher. Are there things that you do at the center level or the organizational level to reduce the burden of the assessments on teachers or on classroom learning, so they can really make sure that they're focused—to your point—on thinking about what students are learning and how they can be supporting them?

Niesha: I believe that our teachers actually appreciate the assessments because they do provide so much rich data, which in turn supports their instruction. It's definitely a guide for their instructional choices. Administering the assessments can be laborious, we know, right? However, it's worth it because, again, they get to tailor the instruction to meet individual student needs right where they are. Teachers want to know what their students are learning, and they want to know that their students are growing.

And from the organization or campus-level standpoint, what I do for my teachers is I print all their assessment protocols for them. I make sure they have all the materials for the assessments prepared already. All they have to do is really focus on administering the assessments. I also make sure that they have coverage in their classrooms during the assessment, so that, again, their main focus could just be on assessing our students.

Abby: And I think one thing to add to that is when we start, Niesha kind of said this, but it is a lift to provide individual direct assessments to children. And when we start working with new schools, or if there's maybe a new partner that's interested in adopting the curriculum model, the instructional model, that is a thing that we'll really point out and look for. Hey, you do need to be able to devote time. You need to be able to devote space to do this well. And you need to provide time for teachers to prepare for teachers to look at the data afterward. So, you do have to be really bought into to the idea that assessment in early childhood is valuable.

Leigh: And you also mentioned that it's really important to make sure that teachers have time to look at the data afterwards. Can you say a little bit more about what that looks like and if there are supports that are provided to teachers for figuring out like, okay, I did this assessment, I learned this about this student, and now I need to figure out how to act on it?

Niesha: At a campus level, we schedule data dives at the end of the assessment window. It's a time where the teaching staff, we all sit together and look at the data together, so that we're able to support one another in making sure we read it properly. We know exactly where our students are. If they've grown, how much progress they have made or have not made. And then what we need to do moving forward.

During the data dives, we look at the data together, but then also develop a plan for moving forward for each student, as well as the classroom as a whole. Because some of the skills you can address whole group. If there's a skill that a lot of students are struggling with, then we'll decide, okay, how can you tackle this skill in your whole-group instruction? So, that's the type of thing that we talk about during the data dives. And it's really very valuable time that's pretty sacred as well.

Abby: And I do think that varies a lot by school. Some schools have one day a week when they close early or one day a month when they're closed. And that might be used for a variety of different professional learning opportunities, but it might be a day to do dive day. For schools that are a little bit newer in working with Every Child Ready and with the assessments in particular, we do often have a dedicated coach or curriculum specialist who might be working...like Niesha has been doing this for a long time.

And I think it's much more fluid for you. We have seen with newer partners, or actually particularly people who are new to early childhood, like pre-K specifically, that it's a shift to think about measurement with three- and four-year-olds and how that impacts instruction. So, we do sometimes have more close contact and heavy coaching support for newer partners. But, again, it really varies based on experience.

Leigh: All right. Niesha, I'm curious to hear from you, what would you say are some of the key strengths of the measurement tools that you're using now?

Niesha: Like Abby mentioned earlier about how it's not just curriculum, it's more wraparound, like how we have professional development as well for teachers. Our teachers receive training on how to use the assessment tool and how to enter the data into the Every Child Ready assessment application. I think that's definitely a strength.

And then also the assessment tool covers a wide range of skills in the core content areas, which is definitely a strength. Again, it addresses the literacy, the language, the math. And this year—and I think the past couple of years—we've been doing the social-emotional learning, which is very big and early childhood education. These are like core skills that students definitely need to be prepared for kindergarten and beyond. That's definitely a strength of the assessments and the measuring tools.

Leigh: And when you talk about measuring social, what are some of what are examples of some of the things that you're looking for in pre-K students?

Abby: I know that in terms of measurement, particularly, but also what's built into kind of the curriculum is that sort of core recognizing emotions in self, others—and then I'll say regulating emotions, I mean, there's so many different ways to think about it, but I think like there's the social piece in terms of how to interact with others, how to be in a classroom, how to be a member of a community and help your peers and help your teachers and advocate for yourself. But then there's also that whole emotion recognition, how our teachers are socializing emotion.

Niesha: Specifically, that the self-regulation for early childhood is so important because students have to be able to regulate their emotions in order to even attend to the instruction. So, just giving them direct instruction on, first, identifying those emotions again, like Abby said, right? But then being able to regulate how you're feeling: why are you feeling sad? Why are you feeling anxious? Because they have to be able to regulate themselves in order to, again, be able to even hear the teacher, be able to even socialize with their peers. For early childhood, that's an actual skill that needs to be taught to them. It's not something that just comes naturally.

Abby: That's such a good point. That's often what you'll see when you start to dig into the classroom data. The first thing people notice is the academic data. You might be like, oh no, these literacy scores are low. But you dig a little bit deeper and it's like, oh, this is actually behavior challenges, this is actually socio-emotional skills. Kids aren't even able to attend to the lesson. They haven't even gotten there at all.

The focus needs to be getting them comfortable in the classroom, feeling like it's a safe learning environment. It's a place that they can be comfortable and relax and be receptive to activities with adults and instruction. And I think we're—I mean, I'm going to make some assumptions about everyone's ages—I think we're all sort of the “you're fine” generation: “You're fine. Just shake it off. Everything's okay.” Which is not super helpful. So, yes, it's like: how does your body feel when you're upset? Does your chest feel tight? Do you feel hot? Helping little kids recognize that is huge so that they can help themselves calm down.

Leigh: Okay. Let's shift a little bit. So, Niesha, can you talk to me a bit about what your schools do to keep parents informed and connected about what their children are learning and what kind of skills they're developing?

Niesha: When you think about it more formally with us talking about assessments, there's time scheduled after each assessment window to hold parent-teacher conferences. Typically, most people know what those look like. But this is the time of the year when the student's individual progress is discussed with the parents. At our particular campus at Parklands, we also hold what we call academic parent-teacher team workshops.

At our workshops, we discuss each student's most recent data, but we also take a deeper dive into one essential skill. If there's a skill that a lot of students are struggling with, we identify that skill and we dive deeper into that with our families. And then we give our parents tips on how to reinforce that skill in the home environment. So, this supports the homeschool connection, but it also accelerates the learning because the parents and the teachers are doing the same thing from school to the home.

Leigh: Great. And when are the workshops held?

Niesha: After the progress monitoring cycles. After baseline data comes back, that's when we set the goals for the year. After we get the baseline data, we sit with our parents and work together to establish goals for their child. And then the midyear and at the end of the year to discuss how much progress the child has made throughout the year.

Leigh: And what does it take, or what do the conversations look like with parents, to help them really figure out how to make meaning out of the specific measures and different things that are coming out of the assessments?

Niesha: That's why we...Honestly, that's why we started the workshops, because they were not understanding the skill. And even like Abby mentioned, you have to get opinions. Everybody's throwing their opinion, right? Parents want this; they want their child to learn this. Teachers want their students to learn something else. But teaching the parents what their child should be learning and what they should know on that developmental progression. Developmentally, what should your child be able to do? What should they know at this age and stage in their life, in their development?

So, just showing parents that. And I think AppleTree, the institute, does a really good job with showing us a progression of skills. And we can actually show the parents what that progression of skills looks like. Like they need to be able to do the syllables before we move to the rhyming. There's a progression of skills that we teach our parents about.

And then that whole piece of what does this look like, I think is the most important piece. Because we can say like subitizing. What does that look like, though? So just breaking down these big words to parent-friendly language and then showing them what that looks like in the classroom. So, when they see the homework come home, they know exactly what that means and how to reinforce those skills in the home environment.

We send things home with them. Like the ones in pre-K, for example, are learning addition. We send home dice—right?—so they can practice, make a game out of it. Or identifying shapes off of the cereal box. Or identifying the letters in your name from a newspaper or a magazine. Just little things like that— letter ID, number ID. Those things are really important, but it's easy to practice those skills just around the home.

So, just making parents more aware of little things they can do. Even on their walk or ride to school, reading the letters on the stop sign. What color is the stop sign? What shape is the stop?

Abby: I think it's also sometimes helpful to think about homework as a family engagement opportunity. During last year when the majority of schools were remote for the whole year, we had to develop a lot of at-home resources. And some of it would just be: have a conversation with your child about our community helpers, because that's what they're learning about right now on Zoom school.

And I think sometimes education and teaching can feel like, oh well, it's addition. And there's, like Niesha was saying, there's subitizing and there's progressions of skills. But when you break things down and you say, ah, just go outside and collect some dirt from the ground and use it to make-believe that you're growing flowers in a garden. That makes it feel much more accessible. And I think because it's coming from your child's school, it feels like you are contributing a lot, but it also kind of reinforces that idea of play as learning. And I think it's easy to kind of move away from that.

And I'm guilty of it myself as a parent. My son will be like, oh, at school today we did—and he'll describe something that sounds like a big game. I'm like: you did one at school? But it's easy to slip away from that and forget, especially for the little kids, for the three-year-olds [that] learning is mostly play at this point with a little bit of direct instruction and guided practice in there.

Leigh: All right. So, Abby, I think this one is maybe one that you could take first. I will leave it to the two of you. What domains of learning would you say are really most important to your mission at AppleTree?

Abby: We would say all of them, but if we had to really focus, and I think what you measure often says what you value, so it would be socio-emotional learning, language, separate from literacy, and then literacy and math. And I do think it's like... Something that's really unfortunately still a big thing is that assessment and program content is still driven so much by accountability.

Most accountability frameworks do not include language. They do not include socio-emotional learning. So, we try to emphasize measurement of socio-emotional learning. We try to emphasize measurement of language. But just looking at our analytic data, those types of assessments get administered with much less frequency than literacy and math. Even though when you talk to teachers, when you talk to school leaders, they're like: oh, socio-emotional learning behavior, that's the most important thing. But that's not the pressure. The pressure that's felt most directly is around literacy and math.

And I don't think that's a unique situation at all. And we're starting to see that focus shift a little bit. But I think it's going to be, unfortunately, a long time until early childhood as a field sees social-emotional learning truly come to the forefront. I think it's there when you talk to people anecdotally, when you look at the research base, it's there, but when you look at measurement, when you look at curriculum, when you look at state accountability, it's not really reflected there.

Leigh: It's a huge credit, I would say, to AppleTree for really figuring out how to make sure that the social and emotional component is also part of what you're assessing.

Abby: Assessment fatigue though is also real. So, I think that makes it really tough. Because I would say I have a physical development background also. Like when I was in grad school, I did a lot with motor development. And I think that's important. But you can't measure everything. And I think what we can do is provide the options for teachers to access those types of things.

In an ideal world, I would love if we had many assessments for every single learning standard that aren't required. But if they're standardized and they're high quality so if someone wants to know more about the fine motor skills of a child, like the visual motor integration of a child, any number of different things like social studies, different types of science, that those resources would be there, but not be required for everyone.

Leigh: And on your point about assessment fatigue, I feel like there may be people listening or people, generally, certainly out there who think, oh, but they're so little these kids, why are you taking up time making them do these individual assessments? They're probably nervous. What are you really learning? Is there anything that you would say to people who are a little bit skeptical about the idea of even being able to really accurately assess a three-year-old or four-year-old?

Niesha: The assessments really are well thought out. And they're set up to not feel threatening even to a child. It's more like playing a game. It's fun, it's interactive, it's engaging. The visuals are fun. The manipulatives are fun. You know how you feel that pressure of, or that anxiety of an assessment, a standardized test? We all, I'm sure, I felt it at some time in our academic career, but they don't really feel that pressure because we go into it like it's a game; it's fun. It's how we frame the assessments for our students that supports them being able to just be relaxed and really show us what they know.

Abby: We would also always recommend that you don't assess kids until they've been in school for a few weeks, because so much of it is rapport and comfort. If it's your first week of school, you don't know that teacher; that is not fun, no matter what. Well, maybe for some kids. But once you're comfortable with the teacher and it's part of your daily routine and you see everyone doing it and they're, like you said, they have manipulatives, they're set up as games, I think that feels different.

That's also why we've gotten some pushback before from schools that are saying: “This is a waste for teachers. Can we hire outside people to do it?” And it's a school decision. If that's what you need to do for staffing, but I would say: no, don't do that because then it does feel like something extra and something different. And you're probably not getting an accurate representation of what a child knows because they don't know that person.

Leigh: And who actually designs the assessments that you use in your classrooms?

Abby: For these particular ones we’re talking about, we make them. That was really an effort to align them with the curriculum, so that they feel useful. A lot of assessments out there, even if they come with a curriculum, don't...     

You might get a score then, like Niesha has 90 and Abby has a 50. And it's like, oh, okay, what do I do now? We really develop these so that people could say, okay, Niesha has a 90, this is the next area we can go with her. Or like, oh Abby has a 50, we need to work backwards and work on some of these more foundational skills.

What we offer in terms of assessment is kind of universal in terms of a school or a state or a district. Because if you're signing on to do the instructional model, you're signing on to do these assessments. Where there is, I think, more differentiation is those ongoing checks for understanding or more like refined informal assessments, all that data still goes into the same sort of data warehouse and produces those same types of reports.

But an individual school could say: you know what? We really just want to do socio-emotional learning. We're just going to pull these Check for Understanding assessment items. And then we'll have a report to show families. We'll have guidance around instruction in that area. So that is an area where schools could tailor it a bit more, but it would really be a decision that a school would make based on their unique needs. That's a really good question.

Leigh: How can programs like yours make decisions about the types of assessment tools they would like to use?

Abby: I would say for early childhood, it is tricky because…if you're a state that has universal pre-K, there are requirements. Like DC has universal pre-K. Very generous for three- and four-year-olds, but that does mean that there's a lot more regulation, which I think is a good thing, around what's being measured and what outcomes are. So, you have to pick an assessment that's also an approved list.

In states with less regulation, or really in a lot of places, you might pick and choose from a variety of different things, but there's huge variability in what could be happening. One of the most popular or frequently used early childhood assessments for pre-K is Teaching Strategies GOLD. It is an observational tool. It's like a teacher report tool.

And the assessments we use are direct one-on-one teacher-administered assessments of children's skills. You've got a set of data that's coming out that's based on direct—I'm asking you a question, you're responding. And then you've got a set of data that's coming out that inform teacher observations. But those are really, really different data sources. And it does make comparisons of programs…it makes universal comparisons across states really difficult because those are very, very different measurement tools.

Now, as you get older, like into third-grade testing, because that's like highs-takes testing, things are pretty standardized then. There's like two or three tests that a state can pick. And that's what everyone does. But early childhood is like the wild, wild west. People could be doing so many different things. They often have to be approved in some way, but they can be vastly different from school to school, depending on what a school has chosen. That can make measurement really tricky.

Leigh: Alright, I just want to say: thanks so much for joining the discussion today, Abby and Niesha. It was really interesting to learn more about your work and the AppleTree Institute.                           

Researchers at MDRC are gathering information about pre-K assessments from folks like today’s guests to think about ways to strengthen pre-K assessments. MDRC’s Measures for Early Success initiative will use these important insights to design and test pre-K assessment tools that are more equitable, more useful to educators and parents, and more effective for informing policy decisions. Stay tuned for more work on this project from MDRC and collaborators.                           

Did you enjoy this episode? Subscribe to the Evidence First podcast for more.