Activity: Reflect on the Pilot

Overview

After you’ve conducted your pilot, it’s time to assess whether it achieved its goals. Design teams use a range of sources to come to this conclusion:

• Feedback from the teachers and students engaged in the pilot

• Data you collect about student learning and experience

• Your own observations and reflections

It can be disappointing to determine that a pilot did not produce your desired results, but do not be afraid to do so – that is the purpose of these bite-sized prototypes and pilots! Figuring out what didn’t work (and why) can help you redesign a better prototype as needed and ultimately helps you avoid scaling something that does not have a positive impact. Remember: pilots are only successful if you learn from them.

Steps to Implementation

Suggested time: 1 hour

1. Make the data from your pilot available to team members, and ask people to review the findings in advance of your meeting.

2. Gather your team in a room (in-person or virtual) with access to the Plan to Pilot template.

3. Remind team members about the purpose/goal of your pilot and how the pilot connects to your broader vision and problem of practice. Indicate that today’s discussion is a chance to evaluate the pilot’s effectiveness in achieving your desired outcomes and to make improvements. Use the Reflect on the Pilot Discussion Questions to guide your conversation.

4. First, review headlines from the student and teacher feedback collected at the end of your pilot. Discuss as a team.

   1. What parts of the prototype resonated most with participants? What were they drawn to?

   2. Which parts resonated least? What made them difficult or confusing?

   3. What should be explored more deeply?

   4. Was the feedback unanimous across participants, or did certain people or groups of people have different experiences with and perspectives on the pilot?

5. Then, review headlines from data produced during the pilot (e.g., observations, a student exit ticket, and/or engagement survey) and discuss your findings as a team.

   1. What went well in the pilot? How do we know?

   2. What did not go well in the pilot? How do we know?

   3. Were the pilot outcomes better or worse than current state or baseline? How do we know?

   4. Was the pilot experience consistently good, or was there meaningful variation in the experience or data (e.g., did all students – especially those we most want to benefit – succeed equally on the student exit ticket, or were there gaps across student groups)?

   5. Were there any unexpected consequences, positive or negative?

6. As a team, summarize your discussion by identifying the top three to five successes and the top three to five challenges of the pilot. You will use this summary perspective in your next step: improving the strategy and building on momentum.

7. Next, you will plan to improve your solution and build on the momentum of your pilot based on your reflections here.

This activity was inspired by the “Get Feedback” step of IDEO’s Design Thinking for Educators Toolkit.

This strategy is a part of TLA's Real-Time Redesign release, a practical toolkit for improving equity and resiliency in schools. Explore the full guide to find additional strategies, insights, and resources.