Contents:

• 2002–03 Data Collection Year Ends
• 2003–04 Data Collection Preparation
• Annual LSC Core Evaluation Requirements

2002–03 Data Collection Year Ends

The 2002–03 data collection year ended August 31.  Any professional development observations conducted after August 31 will be considered part of the new data collection year (2003–04).  In order to close out project responsibilities for the ending data collection year, PIs and Lead Evaluators should complete the requirements outlined in this article.

Principal Investigator Responsibilities

Principal Investigators (PIs) are ultimately responsible for making sure that all core evaluation data are of high quality and are collected and submitted in a timely fashion.  Specifically, PIs should:

1.  Monitor the progress of core evaluation data submission

In order to assist PIs with this task, we have provided a "Data Collection Summary" in the project-specific PI section of the HRI LSC Core Evaluation website so that PIs can track their project's progress toward the October 15 data submission deadline.  It is important to note that forms cannot be submitted after October 15.

In addition to the primary data being submitted by the project (observation and interview protocols, and questionnaire data), the "Data Collection Summary" lists all other information to be submitted by this date (core evaluation report and ratings, and the Project Strategies questionnaire).  All of the individual project information on this summary will be conveyed to NSF on November 15 in a compliance report that they have requested.

2.  Complete a Project Strategies Questionnaire

All projects should complete a Project Strategies Questionnaire.  The form is available for submission on the HRI LSC Core Evaluation website.  Lead Evaluators and PIs should collaborate in preparing responses to the form, but either can submit the form online.  Thus your project's form can be accessed via the project-specific Lead Evaluator and PI sections of the website.  However, it is best to designate only one person to actually enter and submit the data to avoid overwriting each other's changes.

To access the form, log in to your project-specific LE or PI section and select the "Project Strategies Questionnaire" link from the yellow menu at the left of the page.  A complete Project Strategies questionnaire should be submitted to HRI by October 15, 2003.

Lead Evaluator Responsibilities

Lead Evaluators coordinate the submission of all 2002–03 data to HRI.  Specifically, Lead Evaluators should:

1.  Make sure that all core evaluation data required for the project during the 2002–03 data collection year (classroom observations, professional development observations, teacher interviews) are submitted via the Web by October 15

Please make sure that all forms are completed.  Any forms submitted as "partial" will be deleted from our files, so make sure that all forms for your project are finalized and submitted as "complete forms."  You will be unable to submit 2002–03 forms after October 15.  Any forms not submitted as complete will be considered "not collected" by the project.

The online protocols can be accessed by selecting Online Forms from HRI's LSC core evaluation website. A recent standards-compliant web browser (e.g., IE 6.x, Netscape 7.x, Mozilla 1.x, or more recent versions) is recommended for submission of all forms.  When a form has been received (whether as partial so you can return to it later, or complete), a screen confirming that your data have been successfully submitted will be displayed. This page will also display all of your completed responses; please print a copy for your records.  If you do not see the phrase, successfully submitted, your form has not been received at HRI.

2.  Submit a core evaluation report and ratings via the Web by October 15

Again this year, all core evaluation narrative reports and ratings need to be submitted via the Web.  Lead Evaluators should consult the 2002–03 Reporting Guidelines available in the project-specific Lead Evaluator section of the website to determine the type of report required for this year.  There are different versions of the reporting guidelines for projects in their Year One, Year Two, Interim Year and Final Year.

In compiling the core evaluation data collection manual for next year (2003–04), we realized that an edit made to one item on the Year Two and Final Year versions has implications for this year's reporting (2002–03).  In previous years, Year Two and Final Year projects were asked to describe the extent to which three types of federal funds are used by the LSC district(s) to support mathematics/science professional development:

Due to changes in Federal funding options, the types of Federal funds projects are asked to provide the status of have been modified.  The new categories projects are asked about are:

The Year Two and Final Year District Policy Rating Worksheet and Reporting Guidelines for 2002–03 have been edited to reflect this change.  For current copies of these documents, please access your project-specific PI and Lead Evaluator section of the HRI LSC Core Evaluation website and select 2002–03 Data Collection Requirements.

All versions of the reporting guidelines may also be viewed in the online 2002–03 Data Collection Manual.  Using the designated guidelines, the Lead Evaluator should 1) prepare a draft of the 2002–03 Core Evaluation Report; 2) share the draft and accompanying ratings with the PI; 3) address any concerns of the PI; and 4) submit the final version of the report to HRI via the Web.

Tips for Submitting Core Evaluation Reports

There are constraints associated with web-based submission of reports.  To ensure a successful submission of your core evaluation report:

• Create the report in a Word document and cut and paste each response into the corresponding text box.
• Use two asterisks before and after a phrase to indicate emphasis; **do not** use underlining, italics, or bold print.
• Any quantitative data you wish to include will need to be incorporated into the narrative; please **do not** submit charts or tables.
• Indicate quotes with quotation marks; **do not** use italics.

Lead Evaluators will have the option of submitting a "partial" report using the "Submit Partial Form" button and returning to it later for edits and completion.  All completed reports must be submitted as complete using the "Submit Complete Form" button; several checks will be performed at that point to ensure that the report has been completely and consistently filled out.  When a form has been received (whether as partial or complete), a screen confirming that your data have been successfully submitted will be displayed. This page will also display all of your completed responses; please print a copy for your records.  If you do not see the phrase, successfully submitted, your form has not been received at HRI.

3.  Assist PI with completion of the Project Strategies Questionnaire

The Project Strategies questionnaire is required for all projects.  See details in the PI responsibilities section of this article.

Redeeming Classroom Observation Incentive Vouchers

The deadline for redeeming the 2002–03 Classroom Observation Incentive vouchers has been extended until October 22, 2003 Teachers who received a $25 voucher for purchasing materials from the Brain Train catalog and have not yet redeemed it should complete the order form that was given to them and mail it to:

Brain Train
3376 S. Church St.
Burlington, NC 27215

Submitting Invoices for Classroom Observation Incentives

The deadline for submitting invoices for 2002–03 classroom observation incentives is October 22, 2003.  Projects that elected to bill HRI for $25 per completed classroom observation (only those required for the core evaluation count for incentives) should mail their hard copy invoice (faxes are not accepted) to the HRI address listed below.  Please note that if projects prefer the checks be payable to individual teachers, they will need to provide social security numbers.

LSC Classroom Observation Incentives
Horizon Research, Inc.
326 Cloister Ct.
Chapel Hill, NC 27514

2003–04 Data Collection Preparation

Data Collection Schedules

HRI will be communicating with PIs in the next few weeks to confirm LSC project data collection schedules, based on the project end dates.   Project-specific schedules for the 2003–04 data collection year (reflecting any "data collection penalties" imposed by NSF) will be available in the Lead Evaluator and PI sections of the website after the October 15 deadline for protocol submission for the 2002–03 data collection year.

2003–04 Data Collection Manuals

The electronic version of the 2003–04 Data Collection Manual was posted on the HRI LSC Core Evaluation website during the first week in September.  Hard copies of the manual have been mailed out to the Lead Evaluator and PI of each project.  Projects may request additional copies from HRI.

Sampling Frames

Updated school and teacher sampling frames for the 2003–04 Data Collection year are due to HRI by December 2, 2003.  To facilitate this process, HRI has recently provided preformatted sampling frame files, containing data from previous years, for each project to download from the project-specific PI section or Lead Evaluator section of the HRI LSC Core Evaluation website.  Please note that while we are making the files available in both the PI and Lead Evaluator section of the website, the project should submit only one school and teacher sampling frame to HRI. Projects should download the files, update the appropriate information and add any additional schools/targeted teachers, and return the updated files to HRI.  Because HRI uses these files to draw samples for core evaluation data collection activities and because these data are reported to NSF, it is important that these files be as accu­rate as possible, with correct treatment hours for teachers.  Consult the 2003–04 School and Teacher Sampling Frame Guidelines, under Tab 3 of the 2003–04 Data Collection Manual for the correct procedures for updating the files.  These guidelines can also be accessed via the PI/Lead Evaluator sections of the website.

As in previous years, Sampling Frames will be "cleaned" and samples drawn in the order that they are received at HRI.  Projects are encouraged to submit their Sampling Frames as soon as possible in order to begin data collection in early February.

Data Collection Reminder

When adding individuals to the evaluation team, please note the following:

• NSF must approve all members of the evaluation team in advance.  Vitaes of these individuals should be sent to the project's program officer with a note describing the person's proposed role in the evaluation;
• The Lead Evaluator must personally conduct at least three of the required professional development observations each year; and
• Individuals conducting classroom and professional development observations must successfully complete HRI's certification/training process and be approved by NSF prior to participating in the training.

Please notify HRI of any Lead Evaluator/PI assignment changes that may occur during the year. 

Project Team Interviews

Lead Evaluators are reminded that interviews with project staff should be conducted early in the data collection year to obtain information regarding project plans that will help in selecting professional development sessions to be observed.  An interview protocol is available to assist evaluators in collecting this information; it can be found in Tab 8 of the 2003–04 Core Evaluation Manual.

Additional project team interview protocols are provided in Tab 8 to assist evaluators in collecting information that will contribute to answering the core evaluation questions.  After October 15, consult the Project Team Interview Guidelines or the HRI LSC Core Evaluation website for your project's data collection schedule to determine which protocols to use this year.  Please note that these interviews are for Lead Evaluator use only and results should not be sent to HRI.

Supplementing the Core

Next month HRI will be contacting projects to find out if they want to supplement the core evaluation data collection activities by administering "extra" teacher questionnaires.  Projects that are scheduled to administer teacher questionnaires in the spring of 2004 as part of the core evaluation (projects in their Year Two or Final Year of data collection) may request extras if they wish to survey teachers in addition to the random sample selected by HRI, whether that is the rest of the teachers in the project or some particular subgroup.  Projects not scheduled to administer teacher questionnaires in 2003–04 may choose to request them for all of the teachers in the project, a random sample, or a particular subgroup.  HRI anticipates being able to provide extra questionnaires to all projects as long as they are requested by November 3, 2003.  Please note that HRI will provide questionnaires and scanning as part of supplementing the core evaluation, but will analyze and prepare tables only for those projects administering questionnaires to samples of teachers drawn by HRI.

Teacher Incentives

Beginning this 2003–04 data collection year, projects will no longer have the option of obtaining redeemable vouchers from HRI for teacher incentives.  Instead, your project may bill HRI for $25 times n (where n equals the number of completed classroom observations from those required for the core evaluation this spring).  Projects will then be able to use these funds (and additional funds, if they wish) to provide whatever incentives make sense to them for individual teachers and/or schools. Note that teachers included in "supplemental observations" do not figure into the incentive allotment for the project, although projects may decide to provide incentives to them as a matter of equity.

Meeting Plans for 2003–04

The annual Lead Evaluator meeting will be held on January 12–13, 2004 in North Carolina.  Tentatively, the meeting will begin with dinner and an evening session on Monday, and conclude with an 8:30am–3:30pm session on Tuesday.  Please note that attendance at this meeting is at project expense, and is required by NSF.

The Classroom Observation training is scheduled for February 5–6, 2004 in Chapel Hill, NC.   All classroom observers must successfully complete HRI's certification process in order to be eligible to conduct classroom observations for the core evaluation.  NSF has indicated that all observers must have the necessary content background and science/mathematics education experience to conduct classroom observations.  As with all members of the evaluation team, all observers must be approved by NSF; potential observers must be approved by NSF prior to attending the training. 

Specific information regarding these meetings, including details on registration, will be sent out to all projects next month.

Annual LSC Core Evaluation Requirements

All Projects:

In addition, Year Two and Final Year Projects:

Most data collection activities are conducted February–May annually, except Professional Development Observations and the Project Team Interviews, which may be conducted throughout the data collection year (September 1–August 31)

The PI is responsible for making sure that core evaluation data are collected in a timely fashion and reported to HRI each fall for inclusion in the annual cross-site report to NSF.