Contents:

• 2001-2002 Data Collection
• Preparing for the 2002-2003 Data Collection
• Meeting Dates for 2002-2003
• Post-doc Positions Available at HRI

2001-2002 Data Collection

Core Evaluation Reporting

2001-2002 Reporting Guidelines

The 2001-2002 Reporting Guidelines will be available online in read-only format on July 1, 2002 and for Web submission on September 3, 2002. Please note that this year the Core Evaluation Report will be submitted electronically via the Web rather than as a separate narrative report. The deadline for submitting completed 2001-2002 Core Evaluation Reports is October 15, 2002.

Changes in Reporting Requirements for Year Two and Final Year Projects

As introduced at the recent Lead Evaluator meeting, the 2001-2002 Core Evaluation Reporting Guidelines for Year Two and Final Year projects have been revised; requirements for Baseline, Year One, and Interim Years have had minor revisions, but have essentially remained the same.

In past years, Lead Evaluators of Year Two and Final Year projects wrote the Core Evaluation Report responding to a number of questions in the reporting guidelines, describing and assessing the quality of components of the project. In addition, they completed ratings and rationales on the same program areas. This year Lead Evaluators are asked to answer the same general questions (descriptions of project components and assessment of their quality) in an "analytic memo" format submitted online instead of a narrative format. Also, ratings and rationales will be integrated into the analytic memo instead of being a separate task.

When writing this report, Lead Evaluators should keep the following in mind:

• the project approach in each area of the professional development program should be presented in sufficient detail to provide context for the judgments and supporting evidence that follow, but without irrelevant information. See an Example.
• the assessment of the quality of the project’s components should begin with an overall statement of that quality, then provide the major reasons for the judgment; it should include a balanced integration of the most salient interview, observation, and questionnaire data as supporting evidence for the overall statement. See an Example.
• make sure the overall statement of quality is clear and easily distinguished from the supporting evidence.

The links above provide both positive and negative examples of two components, originating from one hypothetical project. While this particular case is a science project, the issues apply to all LSC projects.

To check that responses have the key components, Lead Evaluators should ask the following questions:

• Can a reader tell what is going on in the project and to what extent?
• Does the reader have a sense of the quality?
• Does the reader have confidence in the judgment?

Please note that NSF has indicated that reports submitted to HRI without these components will be returned to the Lead Evaluator for further clarification.

Questionnaire Data and Composites

Data from teacher and principal questionnaires administered in spring 2002 will be available again this year via HRI’s LSC website. Tables for individual items and composites are posted in the order in which project questionnaires are received. The deadline for returning completed questionnaires is July 1, 2002. Please note that those projects administering the short version of the LSC Principal Questionnaire will not have principal composite results, since those items were not included in the questionnaire.

In addition to the data tables and composites, raw data files for both types of questionnaires are also provided. Lead Evaluators and PIs will be notified via e-mail when tables and data files have been made available for their project. Composite reliability figures will be available late in the summer, when all projects have sent their questionnaires for scanning and processing.

Files providing the link between the LSC ID for 2002 and the project teacher ID are available to Lead Evaluators under "Samples and Links" in the project-specific Lead Evaluator section of the LSC website. Teachers and principals are assured of confidentiality when they complete a questionnaire or agree to be observed, so please do not release these links to the PI or project staff.

Monitoring Spring Data Collection

PIs are reminded that they are responsible for ensuring thorough, timely collection and reporting of core evaluation data. The LSC PI can access the PI section of the LSC website by using his or her project-specific username and password. In this section, PIs have access to the project’s data collection requirements; they can link to a data collection summary that will indicate the number of interview and observation protocols the evaluation team has submitted to date, both partial and complete. If you do not have your username and password, please use the "Log-In Help" function on the website, and they will be emailed to you.

Reminder to Baseline and Final Year Projects: District Information

The core evaluation calls for the collection of data on the system the project is attempting to change, as well as on teachers and their teaching. At the beginning of an LSC project, and again at the end, projects are asked to collect and submit information on the context of the districts participating in the LSC; the focus is on policies and practices in these districts, and the amount of support from stakeholders for the reforms the LSC is charged with implementing. PIs of Baseline projects should complete a "District Information: Prior to the LSC" [PDF] form for each participating district, and send copies to HRI and to their Lead Evaluator by August 31, 2002.

PIs of Final Year projects should complete a "District Information: Conclusion of LSC" [PDF] form for each district (or a sample of three districts – see the Data Collection Manual for guidelines on selecting these districts) and send copies to their Lead Evaluator and to HRI. Final Year projects should meet this requirement as soon as possible, if they have not already done so (deadline of July 1, 2002), since their Lead Evaluator will need this information in order to complete the final core evaluation report due to HRI in mid-October.

Copies of both versions of the District Information forms, along with guidelines for completing them, can be found in the Data Collection Manual.

Project Strategies

As part of the reporting process, PIs and Lead Evaluators of LSC projects will be asked to collaborate on describing the design of the project by completing an online Project Strategies form; either the PI or Lead Evaluator can submit and view the form via the LSC website. The Project Strategies form will be available Web submission on September 3, and will be due October 15, 2002.

Preparing for the 2002-2003 Data Collection

Data Collection Requirements: Schedule Confirmation

Once again this fall, HRI will be communicating with PIs in order to confirm LSC project data collection schedules, based on the expected project end dates. All projects collect the same data for the first three years of the project (Baseline, Year One, Year Two); the number of Interim Years of data collection depends on the date the project receives its final funding increment from NSF. Projects have the choice of collecting their Final Year of data either the spring of the year the project is scheduled to receive its final funding increment, or the spring following the last "official" funded spring. Projects will collect their final data during one of these times regardless of whether they are planning to take a no-cost extension at the end of the project.

Many projects choose to delay their final data collection beyond the original funding period in order to increase the likelihood of detecting impact on teachers and their teaching. At the same time, when selecting the final data collection year, projects should keep in mind that they must have enough funding left to complete the full round of data collection in that year, which will likely require evaluation staff to continue to work on core evaluation data collection and reporting through mid-October.

Sampling Frames due in December

Updated School and Teacher Sampling Frames for the 2002-2003 Data Collection year will be due at HRI by December 2, 2002 in order to provide NSF the information they need on teacher treatment levels. To facilitate this process, by mid-September HRI will return the project’s 2001-2002 Sampling Frame files, pre-formatted for the project to update the appropriate fields. Projects should update these fields and submit the Sampling Frames no later than December 2. As in previous years, Sampling Frames will be "cleaned" and samples drawn in the order that they are received. Projects that would like their samples in time to begin Data Collection in early February are encouraged to submit their Sampling Frames as soon as possible in the fall.

Meeting Dates for 2002-2003

Dates have been set for the 2003 Lead Evaluator meeting and training for new classroom observers:

January 13-14, 2003

• Lead Evaluator Meeting

February 6-7, 2003

• Classroom Observation Training

Both meetings will be held in Chapel Hill, NC. Specific information on these meetings will be sent to PIs and Lead Evaluators in the fall.

Post-doc Positions Available at HRI

HRI is accepting applications for one- to two-year post-doctoral positions in research on professional development for science and mathematics teachers. As the cross-site evaluator for the LSC program, HRI has constructed quantitative longitudinal databases with more than five years of data from teachers, principals, and classroom observers. These databases provide unique opportunities to investigate relationships between participation in professional development, school/district factors, and impacts on classroom practice as measured by external observers and teacher self-report. Persons selected for these positions will be expected to collaborate with HRI staff in ongoing research as well as to identify and pursue new areas for investigation. More information about the positions and how to apply is available at http://www.horizon-research.com/postdoc.htm.