Determining success in a sea of challenges

By: Matt Grimes

The Rhode Island Urban Debate League will be entering its 12th year in September, and in many respects, it seems like we’re just starting out.  Sophie and I have surely benefitted from the efforts of student leaders and League directors who have come before us, but the same set of challenges seems to persist every season.  We have several teachers involved in the League who don’t seem to be very interested in teaching debate, preferring to take a back seat to Brown student volunteers–not exactly a sustainable model as students rarely maintain the same free afternoons over the course of a school year much less over four-year Brown tenures.  We have to call students at home to encourage them to come to camp, show up at lunch time to inform the student body of our existence in their school, and repeatedly remind team members of upcoming tournaments.  Teams dwindle over the school year, League administrators and coaches come and go, and the accumulated knowledge of how to teach debate or run  a league seems to graduate every couple years with the student coordinators.  While we spend a lot of time convincing our community partners of the benefits of debate, sometimes my pitches feel hollow in light of these facts.  How do we end this cycle of needing to re-establish our presence in schools every year, of needing to dedicate hours and hours to re-learning the best approaches to League operations, of needing to convince certain potential recruits of the importance of debate instead of having teams flourish on their own? 

These are not easy questions to answer, especially in one summer, so I think it’s important to have realistic expectations of how our project addressing one particular aspect of the League.  Curriculum development has the capacity to address a number of the factors limiting our ability to reach as many students as possible, but can’t tackle them all.  With a revised curriculum, we will hopefully be able to maintain student interest throughout the year, progressing from mastered skill to mastered skill instead of throwing an overwhelming quantity of information from the start.  This would (also? possibly?) improve teacher engagement , giving a better sense of ownership over the activity as a whole that would ideally translate into increased efforts for recruitment and establishing a strong presence in the schools.  For everyone involved (including Brown volunteers), best practices would be documented and new teaching innovations could flourish as basic lessons are already put together.  All of this is great, and if everything goes to plan, we should see some positive signs over the coming school year.  But it won’t be enough to make the program self-sustaining.  Not even close.  

It would be easy to end this summer with a new curriculum packet, call it a day on big RIUDL reforms,  and judge its success or failure based on participation levels, quality of debate, and teacher feedback.  In this scenario, I think we’re bound to judge our project harshly.  We can only make a significant impact when this reform is accompanied by many others, large and small, and clearly improving the League should be our ultimate goal.  The tradeoff is that we won’t be able to tell if the curriculum work is directly, causally  responsible for positive (or even negative) changes in the League.  While it would be great to know this, particularly as it could serve as a model for others, I think we’ll have to be wiling to make this sacrifice.  I’d much rather be confident that the League is in a good place overall and not be able to attribute it to the particular project Sophie and I are working on than know we put out a great curriculum  but see the League stagnate.  Implementing the Starr lessons on program evaluation, however, will be made a little more challenging.

Measurement and Evaluation Boot Camp Notes (4/1/09)

Many thanks to Alison Cohen ’09 for teaching a Boot Camp workshop to help project leaders identify goals & outcomes for their work and begin to develop plans to evaluate their efforts.  The following materials have been provided for your reference by Alison and Professor Marty West who teaches Evaluating the Impact of Social Programs (ED1160).

Marty West – Introduction:  What is Impact Evaluation?    Download

Marty West – Establishing Validity, Understanding Causal Inference, and
Asking the “Compared to What?” Question    Download

Marty West – Developing a Framework for Causal Inferences  Download

Marty West – How Experiments Solve the Evaluation Problem  Download

Alison Cohen – Community-Based ParticipatoryProgram Evaluation: Understanding Key Concepts  Download

Alison Cohen – Causal Pathway Examples  Download

Alison Cohen – Planning Tool Example  Download