Unfinished Learning: What is the process for effectively organizing, analyzing, and making sense of data?

Data can be extremely powerful when used effectively. Unfortunately, that same data can also be misused, mishandled, or incorrectly identified and measured. These three problems can lead to missed opportunities to maximize student learning.

Likewise, relying solely on measuring student learning through summative data versus formative data places districts at risk of not fully understanding students’ skill levels and areas of mastery. Using traditional, summative data to measure student learning can lead to a deficit perspective as comparisons often focus on discrepancies. On the other hand, formative data collected at multiple points in time can allow you to view student learning from the perspective of what improvements have occurred.

Data analysis is important to inform systems about where and how student learning has been optimized, as well as specific areas that need support. Proper data analysis will help you to avoid making false assumptions about student growth. Instead, data analysis and sensemaking can increase the effectiveness and accuracy of communication with stakeholders about student needs. Likewise, accurate data analysis leads to accurate data visualization that provides useful summaries and conclusions for how student performance can be optimized in the future.

• Collecting and organizing the right data is needed to answer research questions. Data can be collected from multiple sources but should be codified according to the questions you are seeking to answer.

• Data analysis enables districts to make data-driven decisions, derive accurate insights, and avoid making decisions that can be costly to student learning.

• Making use of graphs and charts for data visualization makes complex data more understandable. Examples of data visualization tools are Excel, Tableau, and Jamovi.

Good data organization, analysis, and sensemaking are the keys to forming accurate insights about student learning. In turn, these insights will allow you to make more informed strategy decisions when addressing unfinished learning.

Organize the Data

Although it is often overlooked, organizing your data will enable you to measure learner growth in a logical and consistent manner. Once you know what you need to collect, begin pulling data sources into a single online location.

You may have to collect data from multiple sources that will require several data files; for example, demographic data might live in a student information system or human resources database, and assessment data might reside in a platform such as iReady or NWEA MAP. Although it can be a time-consuming process, it is important to gather all of your raw data files into a well-organized and secure location that adheres to your school or district’s student data privacy policy.

Strategies/Resources:

Once your data is organized and secure, you should import it to your chosen spreadsheet or analysis program (such as Excel, Google Sheets, Tableau, or R). The following strategies are useful for organizing and importing data.

Analyzing the Data

Once you have organized your data, you are ready to analyze it. To understand unfinished learning, the goal is to identify patterns of learner growth. Your analysis should examine formative changes over time to gain a better understanding of how students have progressed.

One option is to examine the change in average scores over time. To do this:

• Use your selected spreadsheet or statistical package to calculate the average score for each group and/or subgroup at each point in time for which you have data. (This Google Sheet provides an example in the Sample Analysis tab.)

• Calculate the average percentage of change between each assessment window. Particularly when comparing growth patterns across different assessment types, percentages between scores can be useful for understanding progress.

You may also consider creating and analyzing a frequency table. For each group or subgroup in your sample, a frequency table displays the average score (mean and median), the total number of responses, the standard deviation, as well as the minimum and maximum scores. This can be accomplished using a statistical package or the FREQUENCY command in Google Sheets.

Strategies/Resources:

Data analysis can be approached using a multitude of methods. The key is staying focused on what you want to know. The following strategies can help guide your thinking on data analysis.

Make Sense of the Findings

Now that you have completed your analysis, you are ready to make sense of your findings so that you can connect what you have observed in the data with actionable ideas and strategies that address unfinished learning.

To better understand what you found in your quantitative data, collect additional qualitative data through interviews or focus groups with educators, counselors, leaders, staff, community partners, or students. These stakeholders will be able to provide valuable insights about their experiences and the support they received or provided.

After conducting any interview or focus group, make sure that you debrief immediately with your research team to discuss initial impressions and identify any themes that emerged from the conversation. Later, as you review your transcript, audio, video recording, or notes from the session:

• List out any additional themes, categories, or major ideas.

• Identify important quotes that illustrate key concepts or important points, making sure to properly label each participant quote so that you can connect the feedback with its context.

• Make sure to also look for common responses among all participants as well as varied responses among different subgroups.

As a last step, compare the results from your quantitative data analysis with your focus group/interview data analysis. Use the guiding questions below to make sense of your overall findings.

1. Does the qualitative data support the quantitative data, or is it different?

2. What are the similarities and/or differences between the quantitative and qualitative data?

3. How does the qualitative data provide insight for or explain differences in learning performance among groups of students?

Strategies/Resources:

Making sense of your findings entails connecting your initial goal for measuring unfinished learning to data analysis and finding the relationships between the two. The following strategies can be used to support this process.

Examining unfinished learning takes time; however, the data can be advantageous to your school or district. The pandemic and other disruptions to learning have exacerbated existing academic gaps and created new challenges for all learners. However, by taking an asset-based approach and measuring unfinished learning, you can create both an evidence base and a springboard for maintaining existing enabling systems and developing new systems to support student learning.

Additional Resources and Content:

• Measuring Unfinished Learning: A Guide for Schools and Systems to Understand and Measure Student Progress

• Learning Found, Not Lost: Examining Learner Growth and Supports in LUSD During and Beyond the COVID-19 Crisis

• Unfinished Learning: How do you begin to measure student progress during disruption?