Among many trends, Hedges and Hedberg (2007) found that as more lower-level factors (e.g., student-level) were accounted for in a study’s design—as found in the second, third, and fourth models—the lower the ICC and the less power the designs had to detect effects. Moreover, given the variety of factors that must be incorporated into short-cycle experiments (e.g., specific topic, time of year), this consequence of reduced power is magnified in the case of short-cycle experiments. Consequently, a model with the necessary controls and sufficient power will require a sample size that is unrealistic to achieve for short-cycle experiments. On the other hand, if all necessary controls are not included, it becomes more difficult to trust that effects generalize to a given classroom. Snidjers and Bosker (2004) echo the problem of unrealistic sample sizes when they explain that as within group homogeneity increases, the ICC might increase, but the need for more subjects also increases if hierarchical linear modelling is used (p. 24). Of particular importance to this discussion, their analyses did not include many of the control factors that would most likely be of importance to classroom teachers concerned with the validity and utility of a specific instructional strategy. As mentioned above, these include factors such as a specific topic within a domain, teacher experience with a strategy, student engagement in instruction, and so on. In effect, a large-scale study that was focused on specific instructional strategies would need to include a long list of these other factors to produce findings that are meaningful to individual teachers. The complexity of the design and the sample size required to detect minimally important effect sizes would make it impractical to carry out such a study.
Validity of the Criterion Measure
Another factor on which teacher-conducted experiments compare well to large-scale experiments is the validity of the criterion measure. According to Raudenbush (2005), this issue might be one of the biggest challenges to the validity of the findings from experimental research, particularly as it relates to instructional interventions.
Indeed, one might argue that a failure to attend systematically to this process of creating good outcome measures is the Achilles heel of evaluation research on instructional innovation. If the process is ignored, trivialized, or mismanaged, we’ll be measuring the wrong outcome with high reliability, the right outcome with low reliability, or, in the worst case, we won’t know what we are measuring. If we don’t know what we are measuring, the causal question (Does the new intervention improve achievement?) is meaningless. If we measure the right outcome unreliably, we will likely find a new program ineffective even if it is effective. If we measure the wrong outcome reliably, we may find that the intervention “works,” but we’ll never know whether it works to achieve our goals.
(Raudenbush, 2005, p.29)
Given the short-cycle effect of instructional strategies, it is difficult to find standardized measures of their effectiveness. Most standardized assessments are designed to cover multiple types of knowledge and skill relative to a specific subject area, and focus on more distal outcomes such as student learning. The appropriate criterion measure relative to a given instructional strategy like previewing must be specific to the content and situation for which the teacher engaged students in the strategy. This likely requires using a more proximal outcome than standardized assessments can capture, such as the activation of prior knowledge in the case of previewing. Teacher-conducted research provides for more flexibility and creativity in selecting an outcome measure that is valid in assessing the short-cycle effect of instructional strategies.
The New Paradigm for Teacher as Researcher
The discussion above implies a new paradigm for experimentation involving instructional strategies at the individual teacher level. As described above, that new paradigm will involve new ways to define sampling procedures, generalizability, and validity for criterion measures. That paradigm must also include new interpretations and new parameters for constructs that include:
- Experimental designs appropriate to individual teachers
- Appropriate statistical procedures
- The relevance of outcomes
Experimental designs appropriate to individual teachers
The new paradigm for teacher as researcher is likely to include a small subset of the designs that are available in large-scale research efforts. Specifically, the most common design that can be used by teachers would involve an experimental group of students and a control group of students. Preferably, these would come from the same class and be randomly assigned to experimental and control conditions. The short-cycle effect of specific instructional strategies allows for this situation in that many strategies would be expected to product observable effects in one or two class periods. Those students who were in the experimental condition might first receive instruction with the benefit of the target strategy. While this treatment was occurring, the students in the control group would work independently in some other room (e.g., the library) on learning activities that enhance their knowledge but do not contaminate their prior knowledge of the content. After the one or two class periods, experimental and control groups would switch and the control group would experience the same one or two class period instruction, this time with the benefit of target instructional strategy.
Appropriate statistical procedures
The paradigm for teacher-conducted experiments would also be limited in terms of the statistical procedures that are appropriate. For example, in the Instructional Improvement Cycle (Cherasaro, Reale, Haystead, & Marzano, 2015), teachers are required to have a pre-test and post-test for each student. The pre-test is used to test for baseline equivalence. If experimental and control groups do not demonstrate baseline equivalence, teachers are encouraged to go no further in the experiment. Another common (but less rigorous) statistical approach is to use an ANCOVA design in which the pre-test is used as the covariate and the post-test as the criterion measure.
One consideration for the statistical procedures for teacher-designed experiments is the alpha level (i.e., significance level) used to reject the null hypothesis In large-scale experiments. The alpha level is typically set at less than .05 to adequately guard against type I errors. Such a criterion seems reasonable since the conclusions of large-scale experiments are intended to be generalizable across a broad population. In this context, type I errors could have serious consequences because they magnify the consequences of concluding that an intervention “works” when an effect was detected. However, when an experiment is meant to be generalized to the behaviors of an individual teacher, the consequences of making a type I error are less daunting. Moreover, given the smaller sample sizes in teacher-conducted research, type I errors are less likely to occur in the first place.
The relevance of outcomes
Teacher-designed experiments are much more amenable to focusing on outcomes that are relevant to specific teachers (i.e., those conducting the experiments) than are large-scale experiments on specific instructional strategies. There are multiple reasons for this. For one, teacher-designed experiments are better able to detect the short-cycle effect of specific instructional strategies. Further, such experiments can address specific topics from specific subject areas that are the focus of a small set of lessons. In addition, teachers can include outcomes that directly relate to how they teach or how they might teach. Such outcomes might include student interest, confidence, and level of effort in learning specific topics.
A National Approach
The new paradigm for teacher as researcher must be constructed in an inductive way by conducting multiple teacher-designed experiments and then examining those experiments to glean generalizations about the nature of the practice and recommendations for conducting effective teacher-designed research.
It is feasible to imagine a national effort that consolidates the findings of multiple teacher-designed experiments. Such a national effort might require the following elements:
- A list of instructional strategies culled from research-based instructional models or frameworks
(e.g., Danielson, 2011, 2013; Hattie, 2008; Marzano, 2017)
- Common design specifications (see Instructional Improvement Cycle; Cherasaro, et al., 2015)
- A common data analysis approach (see Instructional Improvement Cycle; Cherasaro, et al., 2015)
- Results sent to common location
- Computation of common effect sizes for the same strategy, then broken down by mediating and moderating variables
(e.g., teacher’s familiarity with the instructional strategy being studied, length of time strategy was employed)
On a small scale, Robert J. Marzano applied this approach and captured findings in an online database of instructional strategies (Meta-Analysis Database of Instructional Strategies, n.d.) From 2004–11, Marzano collected data from more than 500 teachers through action research at 87 schools in 26 districts. The database includes action research studies conducted by 500 teachers on 22 instructional strategies.
Building the New Paradigm for Teacher as Researcher
Ideally, the new paradigm will be built in the context of contemporary approaches to teacher collaboration, including professional learning communities (PLCs) and professional learning networks (PLNs).
Since the 1990s, there has been a growing emphasis on creating structures, processes, routines, and expectations for teachers working together to hone their practice and improve outcomes for students, classrooms, and schools. The PLC movement has resulted in a nearly ubiquitous acceptance of teacher collaboration as an essential ingredient for school improvement. As Hargreaves (2018) notes, the “important debates about collaboration now are no longer about whether it is a good thing or not but about how to undertake it with precise designs that promote inquiry, reflection, better practice and increased commitment to change” (p. xxii).
There is no doubt that instructional change and improvement happens at the individual class and school level. However, school systems cannot achieve sustained improvement in teacher practice and student learning outcomes at any scale if it is solely dependent on the professional prowess of individual classroom teachers. Teacher collaboration has been gaining momentum because teachers’ participation in quality collaborative learning leads to enhanced human capital (knowledge, skills, attitudes), which helps spread change and improvement throughout the system.
Professional Learning Communities
PLCs have roots in the literature on professional collaboration (Rosenholtz, 1991) as well as reflective practice (Schön, 1983; Stenhouse, 1975). The term PLC was used in education in the 1990s (Cuban, 1992; Hord, 1997: Louis, Marks, and Kruse, 1996; McLaughlin, 1993), but it became popular in the first decade of the 21st century (Dufour & Eaker, 1998; DuFour, DuFour, and Eaker, 2008; DuFour, DuFour, Eaker and Many, 2010). The purpose of the PLC model is for educators in schools to operate collaboratively in ways that have a direct, positive influence on student learning.
One manifestation of such collaborative endeavors is for teachers to engage in lesson study. In Japan, lesson study is part of kounaikenshuu, a comprehensive, school-based approach to professional development that formed the crux of school improvement made popular in the United States in the 1990s (Lewis, 2002).
One of the most common components of kounaikenshuu is lesson study (jugyou kenkyuu). In lesson study, groups of teachers meet regularly over long periods of time (ranging from several months to a year) to work on the design, implementation, testing, and improvement of one or several “research lessons” (kenkyuu jugyou). By all indicators, lesson study is extremely popular and highly valued, especially at the elementary level. It is the linchpin of the improvement process.
(Stigler & Heibert, 1999, pp. 110–111)
One might think of the current discussion about teacher as researcher as a more granular manifestation of lesson study as discussed in the later 1990s and early 2000s. PLCs that have engaged in lesson study would likely have easy transition to conducting teacher-designed experiments focused on specific instructional strategies.
Productive collaboration and spread of effective practice at the individual building level is still insufficient for achieving truly sustainable system improvement. For real system improvement, we look to between-school networks—schools helping schools to amplify the benefits and results of effective collaboration. The between-school collaboration approach has been rising in popularity, not as an alternative to school-based PLCs, but as a complement.
Professional Learning Networks
Leveraging the evidence that teacher collaboration leads to improved student and school outcomes (Borko, 2004; Darling-Hammond, 2010; Vescio, Ross, & Adams, 2008), more schools, system leaders, and policy makers around the globe are investing in forms of between-school collaboration to further promote and enhance these desired outcomes (Briscoe et al., 2015; Harris & Jones, 2017; Poortman & Brown, 2018). Networking with other schools and colleagues (e.g., teachers from nearby schools with similar demographics, researchers with teachers) expands the breadth and circulation of evidence-based and practice-grounded knowledge, strategies, and tools that educators can access and use (Lai & McNaughton, 2018).
Types of PLNs. The designs for between-school collaboratives vary in purpose, form, and function. For example, networked improvement communities (NICs) are popular in the U.S and use improvement science methods adapted from healthcare quality improvement to test and scale evidence-based solutions for addressing critical problems of practice (Bryk, Gomez, Grunow, & LeMahieu, 2015). Also gaining popularity in the U.S. are research-practice partnerships, typically collaborations involving one or more schools or districts and research institutions that are intended to stimulate educator learning and school system improvement by increased use of context-specific data and evidence (Barton, Nelsestuen, & Mazzeo, 2014; Coburn, Penuel, & Geil, 2013; Desimone, Wolford, & Hill, 2016; Farrell et al., 2017; Farrell et al., 2018; Henrick, Cobb, Penuel, Jackson, & Clark, 2017; Muñoz & Rodosky, 2015). Other examples of PLNs include knowledge mobilization networks (found in Canada) that are intended to “promote turning evidence-based research into practice” to achieve education improvement (Briscoe et al., 2015; Ng-A-Fook, Kane, Butler, Glithero, & Forte, 2015), and research learning networks (found in England) which are collaborative networks focused on scaling research-informed teaching practice (Brown & Flood, 2020).
These various forms of between-school collaboration are commonly clustered in the school improvement literature under the construct of PLNs. Such networks involve individuals engaged in collaborative learning with others outside of their regular community of practice (e.g., one or more schools, teachers and university researchers) to improve teaching and learning across the school system (Brown & Poortman, 2018).
Evidence base and conditions for success. PLNs are not just a popular school improvement approach. An emerging, and growing, evidence base points to their promise for strengthening teacher engagement and confidence (Owen, 2015; Regilman & Ruben, 2012; Stoll, Bolam, McMahon, Wallace, & Thomas, 2006; Vescio, Ross, & Adams, 2008), teacher instructional practice (Borko, 2004; Dogan, Pringle, & Mesa, 2015; Manfra, 2019; Stoll et al., 2006; Vescio et al., 2008), and student learning (Goddard, Goddard, & Tschannen-Moran, 2007; Manfra, 2019; Stoll et al., 2006; Vescio et al., 2008).
Merely forming a PLN does not guarantee intended results. A variety of conditions can hinder PLN success. For example, changes in the external environment (e.g., funding, leadership turnover) can threaten the long-term viability of PLNs (Hubers & Poortman, 2018), as can too narrow a focus or a direction and activities that do not fully resonate or directly align with members’ pressing needs (Sims & Penny, 2015).
Conversely, several factors tend to support PLN success, such as providing “structured, supported, and properly resourced” opportunities for professional collaboration (Harris & Jones, 2017, p. 22). Based on our experience and informed by current PLN literature (Barletta, et al., 2017; Briscoe et al., 2015; Hargreaves, Parsley, & Cox, 2015; Harris & Jones, 2017; Holdsworth & Maynes, 2017; Poortman & Brown, 2018; Schildkamp, Nehez, & Blossing, 2018; Stoll et al., 2006), we have identified the following PLN enabling conditions:
- Shared vision, goals, and focus
- Collective leadership
- Resources and sustainability supports
- Structured collaboration
- Scientific inquiry and improvement
- Learning and knowledge circulation
Shared vision, goals, and focus. It is critical that PLNs identify a network purpose and outcomes that members find meaningful and worthy of their effort. This vision helps define what members hope to gain from participation and what they hope to achieve as a result.
Collective leadership. PLNs both require and help build shared leadership. They rely on defined structures, processes, and people to set a strategic direction, ensure ongoing management and development of the PLN, and support the day-to-day work of the network. PLN leadership typically involves a combination of formal roles (e.g., steering committee) and informal opportunities (e.g., breakout session presenter). The majority of leaders should come from within (i.e., network members). In addition, for the work to be prioritized and achieve scale, it is important to have leaders contributing from every level—classroom teachers, administrators, other partners.
Resources and sustainability supports. A variety of resources are needed to support and sustain a network’s infrastructure and member participation, including financial, human, and technological resources. For example, networks need resources to support in-person convenings as well as intermediary support to coordinate and manage PLN activities and to support member engagement and follow through. Schools and districts also need to find ways to provide time and other resources (e.g., release time, substitute teachers, travel) for teachers to carry out their collaborative work.
Structured collaboration. It is important for PLNs to establish collaboration structures, processes, activities, and commitments that are aligned to member needs and network goals and that scaffold collaborative learning, problem solving, and co-creation. Structured collaboration promotes action. Examples of supportive structures include: communication routines to promote member interaction; defined network roles (e.g., team facilitators); appropriate collaborative activities (in-person and/or online; frequency of touchpoints); tools to support collaboration (e.g., change frameworks, discussion and data use protocols); and expectations and mutual accountability for member participation.
When networks aligned their structural components with action, they appeared to have further geographical reach, more outputs, an increased number of partnerships, and possibly a greater impact in terms of mobilizing research-based evidence into practice. Specifically, alignment is more than just the existence of network structures and their processes; alignment refers to the ways in which network members come together to create a synergy that moves the network towards achieving its goals.
(Briscoe et al., 2018, p.30)
Scientific inquiry and improvement. To promote meaningful action that leads to change, many PLNs incorporate a central focus on teacher research, systematic data use, and/or cycles of inquiry and improvement. Network members engage in contextually relevant, data-informed cycles of professional inquiry that are aligned to local continuous improvement goals in order to enhance classroom practice and learning. Members work together to learn and tailor research-based strategies and approaches to their specific contexts; experiment with these teaching practices in the classroom; gather local evidence to evaluate the results or intervention effects; reflect, learn, and make instructional adjustments; and repeat the cycle.
To reap the benefits of authentic, teacher-led collaboration, using systematic inquiry is a key mechanism for teams of teachers working in school-level PLCs and between-school networks. Disciplined, reflective teaching practice anchored in sound professional judgment and supported by locally derived evidence promotes change that is internally directed rather than externally imposed.
Teams of educators can attain these high-quality attributes by engaging in systematic investigations into teaching practices, which can take the form of action research, inquiry, lesson study, and improvement science—all slightly different approaches to continuous improvement where educators identify situations where results depart from expectations and then use various forms of structured investigation to understand why and make refinements.
(Barletta et al., 2017, p. 4)
Learning and knowledge circulation. Effective PLN collaboration enables individual teacher and group learning, or what Hargreaves and Fullan (2012) define as professional capital. Networks thrive when new knowledge, tools, ideas, and strategies are circulated throughout the PLN and beyond. Using PLNs to achieve system-wide change requires ongoing, coordinated co-construction and circulation of new knowledge among individuals and collaborative teams that transfer to the school system level. But to achieve scaled effects, network learning and knowledge circulation must extend beyond dissemination. PLN participants need to engage in parallel activities within their own schools, so that colleagues can use and inform new knowledge gained about school and classroom improvement (Brown and Poortman, 2018).
Although each of these conditions is important on its own, PLNs benefit from establishing them as an interdependent, mutually reinforcing set of conditions. Doing so requires PLN leaders to not only focus on the day-to-day activities and desired outcomes of the PLN, but also nurture and strengthen these enabling conditions over time.
Teachers as Researchers within PLCs and PLNs
Because instructional improvement happens within the context of classrooms and schools, we argue that the new paradigm of teacher as researcher can directly enhance PLCs and PLNs. Contextual factors that impact instructional improvement include student and community characteristics (e.g., mobility, languages), school characteristics (e.g., size, locale), and improvement initiatives already underway (e.g., personalized competency based approaches, multi-tiered systems of support, response to intervention).
Using a teacher as researcher approach can provide structure to enhance two of the conditions necessary for PLC and PLN success: structured collaboration and scientific inquiry. Teacher-designed experiments can make the work of PLCs more meaningful by providing a mechanism for teachers to take action on immediate and pressing needs in their school. For example, in a school with a recent influx of Englisher learners, teachers as researchers can test and iterate instructional strategies to support these students right away. The short-cycle nature of teacher-led research can reduce potential lags in adjusting instruction to meet the needs of special populations. Further, using teacher-led research to support ongoing school initiatives, such as response to intervention, can support and enhance implementation without adding activities to teachers’ daily work.
Within PLNs, collaboration centered around teacher-led research can make network activities meaningful across different contexts and promote long-term network sustainability. Highly advanced PLNs organize effectively to create common curriculum materials, engage in lesson study, co-teach, connect students in between-school projects, and so on. But it typically takes concerted effort, time, and commitment for teacher teams within PLNs to get to this level of professional collaboration. A focus on teacher-led research can add significant value in mature networks. At the same time, it can fill a need for new or developing networks. By embedding teacher-led research as a common practice within the network, teachers can benefit right away. As they experience success and build relationships with colleagues, teams can take what they are doing and learning from teacher-led research even further.
Incorporating teacher-led research can also help overcome other common network challenges, such as member turnover and follow through. Unlike collaborative endeavors such as cross-school projects or co-teaching, teams using a teacher as researcher approach are not entirely dependent on others in the group to receive benefit. Teacher-led research can increase team efficiency by allowing members to focus their attention on identifying common problems of practice and deciding how they want to organize their joint work. It can also help teams acclimate teachers who join at different points in time, a common reality that between-school networks face. Embedding a rigorous teacher-led research process can help strengthen PLNs by providing common processes, tools, and routines to smooth the way for teams to do deep inquiry work while leveraging, not impeding, educator needs and autonomy. Repeated use of the same teacher-led research process helps teams develop common language and ways of working together, which further supports team development over time and helps new members more easily join the work.
Strengthening the process of collaboration itself is important but insufficient for achieving a PLN’s goals. Using a common approach to scientific inquiry helps PLN members translate good collaboration into tangible action to improve teaching and learning. By using a systematic approach to conduct classroom experiments, teachers generate local, real-time data to rigorously study the use of research-based strategies with their specific students.
Incorporating teacher-led research as a central organizing feature provides the added benefit of helping PLNs overcome a common, significant challenge related to PLN sustainability—generating data and evidence directly from collaborative work that can shed light on the value and impact of network activities. The data teachers generate through their research are not only immediately relevant and useful for informing day-to-day practice. These data can also be aggregated to help network leaders and funders evaluate network-wide outcomes and assess the relative value of resources dedicated to PLN collaboration. Studying and circulating results from teachers’ short-cycle experiments can also promote collaborative learning around evidence-based instructional strategies and help prevent ineffective practices from spreading within the network. Circulating knowledge within the network about how specific strategies work with which students under what conditions helps scale-up PLN impact.
In summary, the new paradigm for teacher as researcher legitimizes classroom research and re-establishes educators as key agents of change. This approach to research is amplified when embedded in the PLC and PLN collaboration.