Contents

• Student Outcomes Study Plans Due
• Designing Student Outcome Studies
• 2001-2002 LSC Professional Development Observations
• 2001-2002 Spring Data Collection
• Classroom Observation Training Opportunity
• Projects with New Lead Evaluators

Student Outcomes Study Plans Due

As you know, NSF is extremely interested in information about the impact of the LSC on student outcomes; if you haven't received an email about this from your program officer, you should expect one soon.

LSC projects funded since 1999 (Cohorts 5 and later -- see LSC Projects by Cohort (PDF)) are required to conduct studies assessing the impact of the LSC on student outcomes. Projects in Cohorts 5 and 6 should submit their plans for assessing the impact of the LSC on student outcomes no later than March 15, 2002. Westat is available to provide technical assistance in developing these plans to projects that request it. In any event, PIs should refer to the article below "Designing Student Outcome Studies" for the checklist that should be used in formulating their study plans. PIs should submit a copy of their plan to both their NSF program officer and Joy Frechtling at Westat (frechtj1@westat.com) no later than March 15, 2002. Cohort 7 (see LSC Projects by Cohort (PDF)) plans are due to NSF and Westat by November, 15, 2002.

Although student outcome studies were not a requirement for LSCs funded prior to 1999 (Cohorts 1 through 4 -- see LSC Project by Cohort (PDF)), many projects in these cohorts may have already completed student outcome studies or have data that might inform the LSC student outcome component. In the interest of getting the most complete picture of student outcomes in the context of LSC, a representative from Westat will be calling Cohort 1-4 PIs in mid-January to discuss the availability of this information. The interview will be relatively brief (10-15 minutes). A list of interview questions is included below to allow PIs to gather any pertinent information prior to the call. Please direct any questions about this interview to Joy Frechtling at Westat (frechtj1@westat.com).

Interview Questions:

1. Has your LSC collected data on student outcomes? If so, at what level were these collected: student, class, teacher, school district? What instruments were used? How many teachers, classes, or students were included? Approximately how many students were assessed, and what were the grade level(s) at which testing occurred?
2. Have the data been analyzed to determine if the LSC has had an impact on students? Who should I talk to about those analyses?
3. Is there a report available?

If the LSC has not collected any outcome data:

4. Are there student achievement data available at the state or district level that could potentially be analyzed in relation to teacher participation in the LSC? Do you know if these data are kept in such a way that they can be linked to specific teachers?
5. Who would I need to talk to about these data, how they are organized, and the possibility of incorporating them into a study of the impact of the LSC?
6. Would you be able to provide information about the number of hours of LSC professional development each participating teacher had received and when (so student outcomes can be analyzed in relation to teacher participation in the LSC)?
   

Designing Student Outcome Studies

A number of projects have contacted HRI requesting clarification on the NSF requirements for studying the effects of the LSC on student outcomes. Many of these requests deal specifically with what should be included in the study plans that are to be submitted to NSF. While the Handbook for Conducting Studies of the Effects of the LSC on Students (PDF) provides a fairly comprehensive treatment of the issues one must consider when designing such a study, this article attempts to summarize the four main areas that a high-quality plan will incorporate. (While these areas are interrelated and there are many ways one could organize such a plan, this article will follow the order used in the Handbook.) Also, please remember that NSF has contracted with Westat (frechtj1@westat.com) to provide technical assistance to projects as they develop and implement their plans.

I. Instrumentation

A high-quality study plan will describe the specific questions the evaluation is designed to address, what outcomes are going to be studied, how those outcomes will be measured and the level of aggregation (student, class, teacher, school, etc.) that will be used. As part of this description, the plan would include information that makes the case that:

• The studied outcomes are relevant and important to the LSC project and the evaluation questions;
• The chosen instruments appropriately measure the studied outcomes;
• The instruments are reliable; and
• The instruments are potentially sensitive to the LSC treatment.

II. Sampling

Study plans should include a description of the participants in the study, including information on the number and characteristics of students and, if applicable, teachers and/or schools. The plan should provide enough information to allow the reader to assess whether the studied sample is:

• Representative of the population of students being targeted by the LSC;
• Exposed sufficiently to the LSC in order to merit an examination of effect; and
• Large enough to provide statistical power to detect differences in outcomes or to ensure trustworthiness of qualitative methods.

III. Design

There are two elements of the study design that the plan should make clear. First, the plan should describe what comparisons will be made that enable the project to answer the question, "To what extent has the LSC had an effect on student outcomes?" Possibilities include:

• Comparing outcomes for treated students to outcomes for untreated students; and/or
• Comparing outcomes for students with varying degrees of treatment; and/or,
• Comparing outcomes of treated students to another standard (e.g., outcomes of students in similar districts or grade-level equivalent scores).

The plan should also describe how the study will assess the initial equivalency of the comparison groups. Options for examining equivalency, in decreasing order of preference are:

• Using pre and post measures;
• Using a relevant covariate (e.g., reading test scores);
• Using matched samples; or
• Making the case that the samples are initially equivalent, or that an appropriate standard of comparison has been chosen.

IV. Analysis

Finally, the study plan should include a description of what analytic techniques the project plans to employ once the data are collected. Analysis methods and tools should be consistent with the study design and the type and level of outcome data being investigated.

Checklist

Study Plans should include:

• What outcomes will be studied and why;
• How those outcomes will be measured;
• The specific questions the evaluation is designed to address;
• Description of the sample;
• What comparisons will be made;
• How initial equivalency will be assessed and controlled for; and
• How the data will be analyzed.

2001-2002 LSC Professional Development Observations

The 2001-2002 data collection year officially began on September 1, 2001. While most of the LSC data collection occurs in the spring each year, professional development observations take place throughout the data collection year. The following information is summarized from the 2001-2002 Professional Development Observation Guidelines (PDF).

Projects are required to conduct 5-8 professional development observations per year. The Lead Evaluator must personally conduct at least three of the required observations. All members of the project's evaluation team need to be approved by NSF, and all observers of professional development sessions must have been trained either by HRI or by the Lead Evaluator. The Lead Evaluator may request professional development training materials from HRI to conduct training for other approved members of the evaluation team. To obtain these materials, e-mail a request to us at: lsc@horizon-research.com.

Selecting Sessions to Observe

Interviews with the PI and other project team members are a part of the core evaluation (see Project Team Interviews in the 2001-2002 Core Evaluation Data Collection Manual). These interviews should familiarize the Lead Evaluator with both the goals of the LSC professional development and the project design for achieving those goals. These interviews should be conducted early in the data collection year to aid Lead Evaluators in the selection of the sessions to observe.

The most important consideration in selecting the sample of professional development sessions to be observed is making sure to include sessions that address each of the major functions of the LSC (providing teachers an opportunity to deepen their content knowledge; helping teachers learn the appropriate pedagogy to develop students' conceptual understanding of mathematics/science; and helping teachers learn how to use the designated instructional materials and providing support as the teachers implement the materials in the classroom). In many projects, multiple professional development approaches are used to achieve a particular purpose; evaluators should try to observe a variety of these approaches but not at the expense of covering all of the purposes.

Some logistical notes:

• The segment of professional development included on a single observation protocol should not exceed one-half day.
• In cases where a single professional development session consists of multiple break-out sessions, the observer should select one of the breakout sessions and observe it long enough to complete the Professional Development Observation Protocol. Afterward, the observer may wish to circulate to other breakout sessions to gain a more complete view of the nature and quality of the professional development.
• Use the PDOP for all professional development observations selected for the core evaluation, including sessions that are different from the "typical" event-based professional development activity (e.g., a study group, a coaching/mentoring session, a session that targets administrators). As with any observation, complete only those sections and items that are applicable.
• Please note that the Web system will accept only two professional development observations for a given date. If you anticipate the need to submit more than two observations for a particular day, please e-mail HRI (lsc@horizon-research.com) to explain the circumstances prior to attempting to complete the forms on the web.

2001-2002 Spring Data Collection

In previous years, HRI provided samples and questionnaires to projects simultaneously. The samples were made available to the Lead Evaluator when questionnaires were ready to be mailed, which typically occurred in February. In order to maximize the amount of time a project has to conduct interviews and observations, this year HRI will provide observation and interview samples to the Lead Evaluators shortly after the project has submitted "clean" sampling frame files (i.e., there are no problems needing resolution). Questionnaires will be distributed to the designated person as soon as they are available for shipping. We hope that this change in procedure will ease the data collection crunch by allowing more time for Lead Evaluators to schedule and conduct observations and interviews.

Classroom Observation Training Opportunity

A Classroom Observation Training session is scheduled for February 7-8, 2002 in Chapel Hill, NC. Please note that this is the only Classroom Observation Training session scheduled during this data collection year (2001-2002). Projects wishing to have additional observers trained may get more information by sending an email message to lsc@horizon-research.com, containing the names and email addresses of potential attendees.

Though we are initially accepting only two trainees per project, you may register others who will be placed on a waiting list to be accepted as space permits. There is no cost for the training session itself. The cost of travel and accommodations will be the responsibility of the project.

Projects with New Lead Evaluators

NSF has indicated that projects which have hired or anticipate hiring a new Lead Evaluator should plan to send this individual to the New Project Orientation meeting scheduled for the first week of May, 2002. At this meeting, HRI will provide an overall orientation to the core evaluation as well as training in observing professional development and rating overall programs.

If you have hired or anticipate hiring a new Lead Evaluator, please contact HRI with this information as soon as possible so that training requirements can be discussed and contact information can be changed.