Research and measurement can feel like a daunting task even for the most seasoned team. Before beginning, make an informed decision as to whether you are ready to embark on a measurement project. Use the decision chart below to decide whether you have the necessary resources (expertise, time, capacity, and need) to engage in a measurement project.
Decision Tree Activity
Do you have a question that you want to answer or wondering you want to explore? If not, proceed to the resources below. If so, consider the following questions:
Have you ever completed a research study?
Do you have a team who can help?
Do you have time to take on a research project?
If you have answered yes to all of the above questions, you are ready to move to the next step in the guide. If you answered no to any of the questions, you may want to dive into some existing resources. The TLA website provides an abundance of resources to help guide your thinking, beginning with first identifying a problem of practice, and engaging in research and measurement.
No matter the size or scope of your measurement project, you will need:
A defined goal: Every project has a clearly defined purpose. This is your why – the driver of the key questions you would like to explore, the information you want to learn, or the phenomenon you hope to understand. Consider whether the purpose for your inquiry project falls under one or more of the following:
To explain why something might be occurring (e.g., different patterns of student progress; looking at the data)
To explore a trend or phenomenon (e.g., adult wellbeing or student engagement in virtual settings; addressing unfinished learning)
To understand the efficacy of an instructional strategy, new academic program, or piece of technology (e.g., conducting a program evaluation)
To design and implement a measurable solution to a problem (e.g., address math performance gaps among student subgroups)
A means to collect data: Keep in mind that you may already have data available. Many districts and schools have dashboards and/or pre-made reports that provide an overview of student performance. However, you might also need to survey stakeholders, interview students, or observe practices.
The ability and time to analyze and interpret data: This guide will help you navigate through the research process. However, it is very important that you both set aside time for these tasks and make sure that you have the necessary resources at your disposal. Depending on the scope of your project, you might even want to partner with another organization such as TLA to enhance your capacity.
How can I make my project equity-driven?
Once you decide that you are ready to start a project, these steps can help center equity in your work.
Reflect on your own biases and acknowledge historical inequities.
Ensure your research team is diverse and inclusive of the population(s) and subgroup(s) being studied, including race, ethnicity, gender, socioeconomic background, and others.
Use disaggregated data to understand the intricacies among subgroups of students (e.g., students experiencing homelessness, students with learning differences) and adults (e.g., parental status, gender, years of teaching experience).
Just as an architect creates a blueprint to guide the construction of a building, a research blueprint ultimately serves as the foundation for any measurement project. Creating it should be considered an iterative and participatory process as it ultimately guides data collection and analysis.
The research blueprint contains a core set of elements: the purpose, objectives, and research questions. We have created an editable Blueprint Template(note: make a copy of this resource to edit it) to help support your planning.
As you navigate the steps below, we have included a hypothetical scenario based upon TLA’s Creativity Project to illustrate how the pieces of a measurement project all tie together.
Step 1: Determine the Purpose of Your Measurement Project
Every measurement project begins with a purpose: a rationale or reason for engaging in inquiry. The purpose sets the direction for the entire project, guiding the identification of objectives, research questions, data collection, analysis, and interpretation. It describes the WHY driving your project.
Your purpose statement should be big enough that it can be broken into smaller objectives but manageable enough to be measurable. It should also be neutral. For example, to close the achievement gap is too big, and to show that the use of technology improves student learning assumes that you already know that it works. Consider the examples and scenario below as you think about your purpose.
Examples of Purpose Statements
To determine the effectiveness (efficacy) of an after-school math intervention program.
To understand the ecological and leadership conditions surrounding different site-level implementations of a pilot program.
To understand the effects of a cohort experience on systems’ capacity to develop processes that impact equity and broader system priorities.
To pinpoint bright spots in student learning and encompass a diverse view of achievement so that we can scale supports as we prepare students for the future.
Hypothetical Scenario: Creativity Project
Consider the purpose statement below:
The purpose of this study is to measure the effect of providing districts with access to creative tools — for both teachers and students — as well as high-quality professional learning to develop educator skills, capacities, and mindsets to design learning experiences that foster student creativity skills.
This purpose is big enough that it can be broken into multiple parts – but not too big that it becomes unmanageable. While we are looking to explore changes in teacher skills, capacities, and mindsets, the broader purpose is to examine whether the efforts of the district and teachers ultimately influence the development of student creativity skills.
This purpose statement drives a larger project that looks across districts. However, district leaders could use this purpose to look at schools. Building leaders may wonder about variation between classrooms. Teachers or coaches could revise this purpose to understand the effects of their own actions on their students’ creativity skills.
Step 2: Describe the Objectives
The objectives of a measurement project breaks the purpose into manageable parts. They identify WHAT you hope to learn. Objectives should be specific, measurable, and achievable, much like the elements of a SMART goal.
When identifying objectives, think about how they break the purpose statement into smaller parts. Consider…
The stakeholders involved,
The factors or drivers influencing the context, and
The changes that you want to see.
Objective statements begin with action verbs such as examine, evaluate, or understand.
Examples of Project Objectives
These objectives align with the purpose statements from the previous section:
Examine the relationship between student growth in mathematics and homework completion over the past 12 months.
Evaluate the process surrounding the implementation of a pilot program (i.e., frequency of use, quality of experience, type of activity, adherence to plans, responsiveness of students and teachers).
Understand the effects of cohort participation on systems’ capacity to make continuous improvements.
Identify scalable practices for accelerating growth in reading for students who receive free or reduced-price lunch services.
Hypothetical Scenario: Creativity Project
The purpose of this project is to measure the effect of providing districts with access to creative tools for both teachers and students, as well as high-quality professional learning to develop educator skills, capacities, and mindsets to design learning experiences that foster student creativity skills.
This is a broad yet manageable statement, particularly when broken into smaller objectives. Ultimately, the project intends to:
Examine the effects of active participation in professional learning on educators’ beliefs, mindsets, and understanding about fostering student creativity;
Understand students’ perceptions about their participation in learning opportunities designed to foster creativity; and
Examine the effects of engaging in creative learning opportunities on the development of students’ creativity, critical thinking, and creative communication skills.
If the project achieves these objectives, then we will be able to learn enough to attain our purpose. When designing a project, visualize a funnel with a broad purpose statement at the top that will be narrowed down into one or more smaller objectives.
With this example, a district or building leader could certainly design a project to achieve all three of these objectives. Although a classroom teacher could certainly self-assess to understand their own beliefs, mindsets, and understanding, they are more likely to focus on their students’ perceptions and skill development.
Step 3: Identify Your Research Questions
Research questions sit at the heart of a measurement project. These are the specific inquiries that will ultimately allow you to achieve your objectives. Most importantly, research questions guide the collection and analysis of data to uncover insights, attain objectives, and ultimately allow you to fulfill your intended purpose. Well-formulated research questions are clear and concise, specific, relevant, open-ended (not ‘yes/no’ questions), and most importantly measurable.
Another key consideration when developing research questions is to operationalize or define key terms so that all stakeholders share the same understanding. This will also help you to identify appropriate measures such as surveys or questions for focus groups. The following strategy provides guidance around developing clear research questions:
What is the relationship between student growth in mathematics and students’ completion of assigned math homework?
In this example, it is a good idea to first define student growth. If you know what you mean, then you can determine how you will measure it. Will you look at progress based on multiple testing windows using a program such as i-Ready or MAP, the level of proficiency on specific math standards, or some combination? Will you include more near-term, culturally responsive measures such as math identity or math confidence?
The same applies to the term completing. For example, if students complete the homework late, would it still be applicable? What if students only complete a portion of the homework? Clearly defining key terms in your research question will help you, your team, and all other stakeholders stay on the same page during this process.
Here are a few different research questions and notes about how to operationalize key concepts:
To what extent was the pilot program implemented as designed?: This requires you to identify the key components of the program and then decide how to determine whether they were implemented with fidelity.
What influence does cohort participation have on district capacity to make continuous improvement efforts?: Multiple data sources will be required to answer this question. Surveys and focus groups will help you understand the influence of participation. Different questions – as well as the analysis of completed plans – will be used to understand the district’s capacity.
Which scalable practices promote reading proficiency specifically for students who receive free and reduced-price lunch (FRL) services?: Reading proficiency will be determined using both near-term measures designed by teachers and progress, based on multiple testing windows with i-Ready. The data will be disaggregated based on FRL status. To identify practices, coaches will conduct observations to document how teachers are implementing the prescribed reading program.
Hypothetical Scenario: Creativity Project
Research questions translate your objectives into inquiries. For this scenario, each objective corresponds with a specific research question.
RQ1: How does participation in professional learning affect educator beliefs, mindsets, and understanding about their ability to foster student creativity?: This question allows us to look at the relationship between participation in professional learning and educator perceptions.
RQ2: How does participation in learning experiences designed to foster creativity influence students’ mindsets and understanding of creativity?: There are a few assumptions within this research question. First, it assumes that teachers created new learning experiences with the intention of fostering creativity. Second, it assumes that engaging in the experience has an effect on students’ mindsets and understanding.
RQ3: What is the effect of engaging in creative learning experiences on the development of students’ creativity, critical thinking, and creative communication skills?: This last question allows us to look at specific student outcomes that would theoretically result from engaging in the creative learning experience.
Notice that each research question matches the objectives and that the questions all build toward the broader purpose. As will be explained in the following sections, how the questions are worded then drives the type(s) of data that need to be collected and the way(s) in which that data should be analyzed.
Similar to our objectives example, a district or school leader could address all three questions, but a classroom educator would focus more specifically on their students.
With the first three steps of your blueprint completed, you are now ready to determine your data sources (Step 4) and define your data analysis plan (Step 5). The next section of the guide will help you to collect, analyze, and communicate about your data.
