Contents:

• January Lead Evaluator Meeting
• 2003-04 Data Collection Requirements
• 2003-04 LSC Professional Development Observations
• 2003-04 Spring Data Collection
• Classroom Observation Training
• Projects with New Lead Evaluators

January Lead Evaluator Meeting

The annual Lead Evaluator meeting will be held on January 12−13, 2004 at the Homewood Suites in Durham, NC.  It is scheduled to begin on Monday with a 10:00−5:00 session followed by dinner (optional), and will conclude with a 9:00−12:00 session on Tuesday.  This year the meeting will focus on the structure of core evaluation reporting, case study opportunities available to LSC evaluators, and interrater reliability of professional development observations.  Please be reminded that attendance at this meeting is required by NSF.  Lead Evaluators are asked to plan their travel arrangements so that they attend the entire meeting.

2003−04 Data Collection Requirements

Project-specific core evaluation requirements for the 2003−04 data collection year are available on HRI’s LSC website.  To access these requirements, Lead Evaluators and Principal Investigators should log into their project-specific LE or PI section of the website and select "Data Collection Requirements" from the yellow menu at the left. Please note that the list of requirements includes any acquired penalties for data not collected from the 2002−03 data collection year.

2003−04 LSC Professional Development Observations

September 1, 2003 was the official start of the 2003−04 data collection year.  Although most of the LSC data collection occurs during the spring each year, professional development observations take place throughout the data collection year.  Summarized below is information from the 2003−04 Professional Development Observation Guidelines (PDF).

Projects are required to conduct 5−8 professional development observations per year.  The Lead Evaluator must personally conduct at least three of the required observations.  All members of the project’s evaluation team must be approved by NSF, and all observers of professional development sessions must have been trained either by HRI or by the Lead Evaluator.  The Lead Evaluator may request professional development training materials from HRI to conduct training for other approved members of the evaluation team.  To obtain these materials, e-mail a request to lsc@horizon-research.com.

Selecting Professional Development Sessions to Observe

As part of the core evaluation, Lead Evaluators are required to conduct interviews with the PI and other project team members.  These interviews should familiarize the Lead Evaluator with both the goals of the LSC professional development and the project design for achieving those goals.

When selecting the sample of professional development sessions to be observed, it is important to include sessions that address each of the major functions of the LSC (providing teachers an opportunity to deepen their content knowledge; helping teachers learn the appropriate pedagogy to develop students’ conceptual understanding of mathematics/science; and helping teachers learn how to use the designated instructional materials and provide support as the teachers implement the materials in the classroom).  In many projects, multiple professional development approaches are used to achieve a particular purpose; evaluators should try to observe a variety of these approaches, but not at the expense of covering all of the purposes.

Some logistical notes:

• The segment of professional development included on a single observation protocol should not exceed one-half day.
• In cases where a single professional development session consists of multiple break-out sessions, the observer should select one of the break-out sessions and observe it long enough to complete the Professional Development Observation Protocol (PDOP).  Afterward, the observer may wish to circulate to other break-out sessions to gain a more complete view of the nature and quality of the professional development.
• Use the PDOP for all professional development observations selected for the core evaluation, including sessions that are different from the “typical” event-based professional development activity (e.g., a study group, a coaching/mentoring session, a session that targets administrators). As with any observation, complete only those sections and items that are applicable.
• Please note that the Web system will accept only two professional development observations from a project for a given date.  If you anticipate the need to submit more than two observations for a particular day, please e-mail HRI (lsc@horizon-research.com) to explain the circumstances prior to attempting to complete the forms on the Web.

Submitting Professional Development Observations on the Web

December Lead Evaluators are now able to submit 2003−04 Professional Development Observations via the Web.  The online observation protocol is available by selecting "Online Forms" from HRI’s LSC website.  Please note that all online forms are password-protected. 

Lead Evaluators are reminded to go to their project-specific LE section of the website and update their project's list of trained LSC professional development observers who are conducting observations for the project (e.g., add individuals who have been trained and remove those who no longer work on the project).  Professional Development Observation Protocols will not be accepted as complete forms unless observers have been designated from this list.  

When a complete form is submitted, several checks will be performed to ensure that the form has been completely and consistently filled out.  If necessary, you will be asked to correct problems and submit the form again.

Please note that when a form has been received (whether as partial or complete), a screen confirming that your data have been successfully submitted will be displayed.  This page will also display all of your completed responses.  Please print a copy for your records. If you do not see the phrase "successfully submitted," your form has not been received and your data will not be saved.

2003−04 Spring Data Collection

Samples

December 2, 2003 was the deadline for submitting updated sampling frame files for the 2003−04 data collection year.  For those projects that are still working with HRI to clean their files, it is essential that those files be submitted as soon as possible in order to allow a sufficient amount of time to schedule and conduct observations and interviews. Updated sampling frame files should be submitted via e-mail attachment in tab-delimited text format.

Core evaluation samples for the 2003−04 data collection year are being drawn and will be made available, along with the ID link files, to Lead Evaluators via HRI’s LSC website, provided the sampling frame files for the project are "clean."  Please note that for some projects these samples include "penalties" for data not collected in the 2002−03 data collection year. Lead Evaluators can access their core evaluation samples by logging into their project-specific LE section of the website and selecting "Samples and Links" from the yellow menu at the left.  Evaluator-specific username and password are required; if you don’t have these, use the "Login Help" option.

Since samples and links are made available to Lead Evaluators via the Web, all Lead Evaluators are reminded that teachers participating in the core evaluation (by completing questionnaires or agreeing to be interviewed or observed) are assured of anonymity.  Please do not to share your evaluator-specific username and password with any individual. 

Questionnaires

Principal and teacher questionnaires for the 2003−04 data collection year have been ordered and should arrive at HRI by late January, at which time we will package and ship them to projects in the order in which we received clean sampling frames.  To determine the version of principal questionnaire your project is scheduled to administer, please consult your project-specific data collection schedule on HRI’s LSC website.  Projects are reminded to e-mail HRI (lsc@horizon-research.com) the name, address, and phone number of the individual to whom the supply of questionnaires should be shipped.

Change to Classroom Observation Incentive Plan

Once again NSF has approved the use of core evaluation funds to provide incentives to teachers who agree to participate in the classroom observation component of the evaluation.  However, this year projects will no longer have the option of obtaining redeemable vouchers from HRI.  Instead, projects may bill HRI for $25 times n (where n equals the number of completed classroom observations from those required for the core evaluation this spring).  Projects will then be able to use these funds (and additional funds, if they wish) to provide whatever incentives make sense to them for individual teachers and/or schools.  Note that teachers included in "supplemental observations" do not figure into the incentive allotment for the project, although projects may decide to provide incentives to them as a matter of equity.

Classroom Observation Training

The only opportunity for certification in classroom observations during this 2003−04 data collection year is scheduled for February 5−6, 2004 in Durham, NC.  Projects wishing to have additional observers trained may get more information by sending an e-mail message to lsc@horizon-research.com, containing the names and e-mail addresses of potential attendees by January 5, 2004.  There is no cost for the training session itself for individuals who will be conducting observations for the core evaluation.  The cost of travel and accommodations will be the responsibility of the project.

NSF has indicated that all observers must have the necessary content background and science/mathematics education experience to conduct classroom observations.  As with all members of the evaluation team, all observers must be approved by NSF; potential observers must be approved by NSF prior to attending the training.  Notification of approval should be e-mailed to HRI.

Projects with New Lead Evaluators

NSF would like to remind projects that anticipate hiring a new Lead Evaluator to obtain NSF approval first.  They should then plan to contact HRI with this information as soon as possible so that training requirements can be arranged and contact information can be updated.