March ’97

LSC Core Update

Feature: Observation Training

Evaluators from all of the current LSC projects attended training sessions during February and March aimed at increasing interrater reliability on the classroom observations. Observers had the opportunity to view videotapes of science and mathematics classes at each of a number of levels on the LSC rating scale. Each videotaped lesson was rated on a number of indicators in each of four component areas: design, implementation, content, and culture/equity. Observers then assigned each lesson a "capsule rating":

One important revision to the ratings scale was to subdivide Level 3 into ratings of "low 3," "solid 3," and "high 3"; this change was made to enable projects to more readily demonstrate progress as teachers make improvements within the 3 range.

The following graphs demonstrate a likely pattern for the LSCs. Prior to the LSC program, one might expect lessons to be at a variety of levels on the rating scale, with many in the 2 to low 3 range. By the end of the initiative, one would expect to see movement to the higher end of the scale; while many lessons would still be in the 3 range, they might be high rather than low 3. In the hypothetical example shown below, only 12 percent of the lessons were high 3 and above prior to the LSC, while 77 percent were high 3 and above at the conclusion of the initiative.

We expect Level 5 ratings to be extremely rare both before and after LSC, for a number of reasons. First, in some cases it may be difficult to distinguish between Levels 4 and 5; a really excellent teacher may be so adept at modifying a lesson in response to student needs that neither the teacher nor the observer is aware of the process.

Second, Level 5 instruction may require more than a well-designed lesson, a highly knowledgeable teacher, and competent, effective implementation; it may also require a flair for teaching. At one of the training sessions we likened it to the "artistic expression" rating used with Olympic figure skaters. Level 5 can be thought of as a Level 4 that is so artful you want to applaud the "performance."

Some other key points emphasized in the training:

		The key to judging a lesson is whether it helps develop the conceptual understanding of students and move their thinking forward.
		The balance of content and process and the degree of teacher-directedness need to be appropriate for the purposes of the lesson.
		Inquiry is not just "messing around"; while open exploration may be appropriate early on, at some point structured inquiry is necessary to move conceptual understanding forward.
		An engaging culture and an equitable class can increase the effectiveness of a lesson, but they cannot substitute for a well-crafted learning opportunity.
		Observers need to pay particular attention to the lesson "wrap-up," a place where many otherwise effective lessons break down; the degree of closure/resolution needs to be well-implemented and appropriate for the purposes of the lesson.

The final activity at the training session was to have each observer rate two or three videotapes independently to be used as a check on interrater reliability. We are pleased to note that there have been major improvements in overall interrater reliability since the October 1996 PI/Evaluator meeting in North Carolina.

A few observers tend always to rate too high, or too low, and need to recalibrate themselves. HRI is working with them on that. In a few cases the problem appears to be more serious than simply lack of calibration. A number of evaluators who do not have a strong content background are finding it difficult to judge the quality, significance, accuracy, and age-appropriateness of science and mathematics instruction. NSF is considering ways to address this problem, including asking the project’s lead evaluator to delegate the classroom observations to someone with content expertise. At the very least, observers who lack content background should go over their field notes with someone with content expertise to validate their interpretations and conclusions. NSF has also asked HRI to schedule an additional training session for April so that projects that wish to substitute or add observers can do so.

FYI: HRI’s Web Site

HRI’s web site has been up and running for the past two months, but is still very much under construction. The HRI web page address is:

http://www.horizon-research.com

The URLs (web addresses) for various online forms may change as the site develops, so we recommend that you do not bookmark any page other than HRI’s main home page.

The web addresses for online forms were listed incorrectly in the guidelines found under tab 5 (classroom observations), tab 6 (teacher interview summary), and tab 7 (professional development observations) in the 1997 Data Collection Manual.

To access any online LSC forms, direct your web browser to the HRI web page (http://www.horizon-research.com); from there, make the following selections:

PROJECTS

NSF’s Local Systemic Change through Teacher Enhancement Initiative

LSC Online Forms

You will then find a list of the online LSC forms and you can select the form you wish to use.

Note that the web browser you use will have an influence on the appearance of the HRI online forms. Some features on our site will not be available unless you are using an up-to-date web browser. The site is best viewed using Netscape Navigator 3.0 or higher, and you must be using Netscape Navigator 2.0 or higher to use the on-line forms.

Annual LSC Core Evaluation Requirements
	Professional Development Observations (5-8 per year)
	Teacher Questionnaires (300 if one subject; 450 if two subjects)
	Principal Questionnaires (population)
	Classroom Observations (10 per subject)
	Individual Teacher Interviews (10; not conducted baseline data collection year)
All data collection activities are conducted March-May annually, except professional development observations which are conducted throughout the data collection year (September 1 through August 31).
Project evaluators are responsible for reporting these data to HRI each fall for inclusion in the annual Cross-Site Report prepared by HRI.

Keep in mind that different browsers have different capabilities (for example, you cannot print a copy of the completed form with your entered data if you use Netscape Navigator 2.0 or America Online’s Web Browser, but you will be able to print your data if you are using Netscape 3.0 or Microsoft Internet Explorer 3.0).

Unfortunately the text boxes for open-ended questions in the online forms are of a set size so that if you are entering lengthy open-ended data, you will not be able to print out your entire response even if you are using the most up-to-date browser (the data will, however, be sent to HRI in their entirety).

Depending on the speed of your Internet connection, both the Classroom and Professional Development Observation Protocols could take a few minutes to open up in your browser. Please allow the forms to open completely before you begin filling them out.

It is important that you allow ample time for the data entry, as the online forms must be completed in one sitting.

Due to its size, the Professional Development Observation Protocol is split into two sections on the web. After completing Section I of the form you will be sent to Section II; please be sure you fill out both sections for each professional development observation.

Thank you for your patience as we refine the HRI web site. If you experience any problems using the online forms or have any questions about the web site please contact Gene Soar at HRI. He can be reached by phone at 919-489-1725 or through e-mail at lsc@horizon-research.com.

Something New: Observation Protocols

Since the core evaluation data collection year runs from September 1 through August 31, evaluators may have conducted some professional development observations on older forms prior to receiving the 1997 Core Evaluation Data Collection Manual that contained a revised version of the Professional Development Observation Protocol.

There are two options for evaluators who have observations for 1996–97 that were conducted using the old form:

Transfer the observation yourself to the new protocol, submitting it to HRI via the Web or on a scannable version of the form.

Send the forms to HRI by April 15. HRI will transfer your observation to the new version of the protocol and provide you with re-ratings for your review.

Note that these options apply only to professional development observations conducted between September 1, 1996 and January 31, 1997 using the older version of the protocol. All observations conducted after January 31, 1997 must be submitted to HRI either via the Web page or on a scannable version of the 1997 protocol.

FYI: Extending the Core

A number of projects have received supplemental funding to collect additional data to inform their projects and enrich the cross-site evaluation report.

Additional Classroom Observations

Six projects have been funded for classroom observations beyond the required core random sample this spring. They are:

Atlanta, GA: ESEP (Robert DeHaan, PI) The supplemental funding will enable the project to conduct a set of observations in lead teacher classrooms.

Fresno, CA: TESS (Georgina Takemoto, PI) The HRI contract will fund approximately 50 classroom observations the project plans to conduct with science and mathematics teachers at various stages of LSC participation.

Pittsburgh, PA: ASSET (Reeny Davison, PI) Supplemental observations will focus on lead teachers, trainers, and teachers at various stages of LSC participation.

Rochester, NY: MMRR (Raffaella Borasi, PI) In addition to baseline observations for the core, this project will conduct a set of observations with teachers who have participated in their intense professional development which began last summer.

Santa Barbara, CA: PRIME (Julian Weissglass, PI) New teachers, experienced teachers, and lead teachers will be observed to document the reform on teachers with varying backgrounds; sample will include bilingual teachers.

Santa Cruz, CA: LASERS (Patricia Stoddart, PI) Additional observations will focus on teachers participating in the project’s second tier, who receive training and support from the project’s trained resource teachers.

School Documentation

Another set of projects is receiving supplemental funding to document how their LSC initiatives play out at the school level; while specific plans vary by site, most are using interviews, observations, and collection of school and classroom documents to tell the story of two or more schools in their project.

Projects funded for school documentation include:

Atlanta, GA: ESEP (Robert DeHaan, PI)

Fresno, CA: TESS (Georgina Takemoto, PI)

Nashville, TN: Metro Nashville (Barbara Nye, PI)

New York, NY: SMART Process (Gilbert Turchin, PI)

Santa Barbara, CA: PRIME (Julian Weissglass, PI)

West Windsor, NJ: E=MC² (Sondra Markman, PI)

Researchers will compare notes about effective methodologies at the next PI meeting to determine if there are some components of the school documentation that should be routinely included.

Thank you to all the projects who submitted applications for supplemental funding. We look forward to seeing the results of these projects next fall.

Core Evaluation Q&A: Rating Categories

Send us your questions about the LSC core evaluation system. Any questions of general interest to the LSC community will be answered in this newsletter.

Q: I compared the rating categories in the reporting guidelines in the 1997 Data Collection Manual with the corresponding rating categories on the 1997 Classroom Observation Protocol. Can you explain why there are some discrepancies?

A: The reporting guidelines contained under Tab 10 in the current manual are the guidelines used by evaluators for the 1995–96 core evaluation reports. Evaluators will be provided with updated reporting guidelines based on the 1997 data collection instruments for the core evaluation reports due in the fall of 1997 (covering the period from September 1, 1996 - August 31, 1997).

Q: We want to know if staff (e.g. Project Directors) from other LSC projects can serve in observer roles in our LSC? Is it possible to exchange observer roles? Is there any conflict of interest?

A: NSF has decided that this is fine for projects who wish to do this. Staff from other projects can conduct professional development observations for other LSC projects; however, they can only conduct classroom observations if they have participated in an HRI training session (training will be offered annually for projects interested in getting additional people trained). It is important to note that PIs should not conduct core evaluation observations for their own projects.

Q: Our project has had difficulty in gaining access to classrooms of teachers who have not yet participated in our project. We would prefer to conduct classroom observations with "treated" teachers because we believe we’ll have better participation. Is this possible?

A: This year, HRI is drawing classroom observation samples randomly from the treated teachers in the data collection sampling frame. In addition, HRI will also provide an alternate for each sampled teacher. The alternate is also a treated teacher, usually at the same school as the originally sampled teacher (to facilitate access for observers who have already contacted the principal). Of course, projects collecting baseline data will still need to conduct classroom observations with untreated teachers; HRI will provide a sample for baseline data collection that is selected randomly from the entire data collection sampling frame.

Dates to Remember

03/01–05/31/97: LSC Core Eval Spring Data Collection Activities

03/20–03/22/97: New project meeting at NSF

06/20/97: Completed questionnaires due at HRI for processing

08/31/97: HRI sends questionnaire results to PI and evaluator

08/31/97: End of 1996-97 core evaluation data collection year

10/01/97: Evaluator sends evaluation report to PI for feedback

10/15/97: Evaluator sends evaluation report to HRI

11/15/97: PI sends progress report to NSF with copy of evaluation report

HRI Requests and Reminders: Please Note

		Reminder: Data Collection Sampling Frames should be sent to HRI at least 30 days prior to the date you want to receive questionnaires and samples (see tab 2 in the manual for detailed guidelines). It is very important to submit your sampling frame in the format detailed in these guidelines. Projects will be asked to resubmit sampling frames if they do not contain the requested information in the correct format.
		HRI will provide observation training session participants with information about ordering copies of the videotapes used at the training. This information will be mailed with other post-training session materials to all participants.
		Our new email address for LSC-related correspondence is lsc@horizon-research.com.

[Home] [Projects] [Staff] [Services]