Contents:

• 2001-2002 Data Collection Year Ends
• 2002-2003 Data Collection Preparation
• New Contact at HRI
• Annual LSC Core Evaluation Requirements

2001-2002 Data Collection Year Ends

The 2001-2002 data collection year ended August 31.  Any professional development observations conducted after August 31 will be considered part of the new data collection year (2002-2003).  In order to close out project responsibilities for the ending data collection year, PIs and Lead Evaluators should complete the requirements outlined in this article.

Principal Investigator Responsibilities

Principal Investigators (PIs) are ultimately responsible for making sure that all core evaluation data are of high quality and are collected and submitted in a timely fashion.  Specifically, PIs should:

1.  Monitor the progress of core evaluation data submission

In order to assist PIs with this task, we have provided a "Data Collection Summary" in the project-specific PI section of the HRI LSC website so that PIs can track their project’s progress toward the October 15 data submission deadline. It is important to note that forms cannot be submitted after October 15.

In addition to the primary data being submitted by the project (observation and interview protocols, and questionnaire data), the "Data Collection Summary" lists all other information to be submitted by this date (core evaluation report and ratings, and the Project Strategies questionnaire).  All of the individual project information on this summary will be conveyed to NSF on November 15 in a compliance report that they have requested.  PIs can access their compliance information by logging in to their project-specific PI section of the HRI LSC website and selecting "Data Collection Summary" from the yellow menu at the left of the page.  If you do not have your username and password, click on "Log-in Help" and they will be emailed to you. 

2.  Complete a Project Strategies Questionnaire

All projects, with the exception of those in the Baseline Year of data collection, should complete a Project Strategies Questionnaire.  The form is available for submission on the HRI LSC website.  Lead Evaluators and PIs should collaborate in preparing responses to the form, but either can submit the form online.  Thus your project’s form can be accessed via the project-specific Lead Evaluator and PI sections of the website.  However, it is best to designate only one person to actually enter and submit the data to avoid overwriting each other’s changes.

To access the form, log in to your project-specific LE or PI section and select the "Project Strategies Questionnaire" link from the yellow menu at the left of the page.  A complete Project Strategies form should be submitted to HRI by October 15, 2002. 

Lead Evaluator Responsibilities

Lead Evaluators coordinate the submission of all 2001-2002 data to HRI.  Specifically, Lead Evaluators should:

1.  Make sure that all data required for the project during the 2001-2002 data collection year (classroom observations, professional development observations, teacher interviews) are submitted via the Web by October 15

Please make sure that all forms are completed.  Any forms submitted as "partial" will be deleted from our files, so make sure that all forms for your project are finalized and submitted as "complete forms."  You will be unable to submit 2001-2002 forms after October 15.  Any forms not submitted as complete will be considered "not collected" by the project.

The online protocols can be accessed by selecting Online Forms from HRI’s LSC core evaluation website. A recent standards-compliant web browser (e.g., Internet Explorer 5.x or 6.x, Netscape Navigator 6.x) is recommended for submission of all forms.  When a form has been received (whether as partial so you can return to it later, or complete), a screen confirming that your data have been successfully submitted will be displayed. This page will also display all of your completed responses; please print a copy for your records.  If you do not see the phrase, "successfully submitted", your form has not been received (so your data will not be saved). 

2.  Submit a core evaluation report and ratings via the Web by October 15

Beginning this year all core evaluation narrative reports and ratings need to be submitted via the Web.  Lead Evaluators should consult the 2001-2002 Reporting Guidelines available in the project-specific Lead Evaluator section of the website to determine the type of report required for this year.  There are different versions of the reporting guidelines for projects in their Baseline, Year One, Year Two, Interim Year and Final Years. 

To access the reporting guidelines, Lead Evaluators can log into their project-specific Lead Evaluator section of the website, go to "Data Collection Requirements," scroll down to the "Core Evaluation Report" section and click on the guidelines listed.  All versions may also be viewed in the online 2001-2002 Data Collection Manual.  Using the designated guidelines, the Lead Evaluator should 1) prepare a draft of the 2001-2002 Core Evaluation Report; 2) share the draft and accompanying ratings with the PI; 3) address any concerns of the PI; and 4) submit the final version of the report to HRI via the Web. 

3.  Assist PI with completion of the Project Strategies Form

The Project Strategies form is required for all projects except for those in their Baseline Year.  See details in the PI responsibilities section of this article.

Tips for Submitting Core Evaluation Reports

There are constraints associated with web-based submission of reports.  To ensure a successful submission of your core evaluation report:

• Create the report in a Word document and cut and paste each response into the corresponding text box.
• Use two asterisks before and after a phrase to indicate emphasis; **do not** use underlining, italics, or bold print. 
• Any quantitative data you wish to include will need to be incorporated into the narrative; please **do not** submit charts or tables.  
• Indicate quotes with quotation marks; **do not** use italics.

Lead Evaluators will have the option of submitting a "partial" report using the "Submit Partial Form" button and returning to it later for edits and completion.  All completed reports must be submitted as complete using the "Submit Complete Form" button.   Several checks will be performed to ensure that the report has been completely and consistently filled out.  When a form has been received (whether as partial or complete), a screen confirming that your data have been successfully submitted will be displayed. This page will also display all of your completed responses; please print a copy for your records.  If you do not see the phrase, "successfully submitted", your form has not been received and your data will not be saved. 

Redeeming Classroom Observation Incentive Vouchers

The 2001-2002 Classroom Observation Incentive vouchers for purchasing materials from the Novostar catalog must be redeemed by October 1, 2002.  Teachers who received a $25 voucher should complete the order form that was given to them and mail it to:

Novostar
PO Box
1328 Burlington, NC  27716-1328

Submitting Invoices for Classroom Observation Incentives

The deadline for submitting invoices for 2001-2002 classroom observation incentives is October 31, 2002.  Projects that elected to bill HRI for $25 per completed classroom observation (only those required for the core evaluation count for incentives) should mail their hard copy invoice (faxes are not accepted) to the HRI address listed below.  Please note that if projects prefer the checks be payable to individual teachers, they will need to provide social security numbers. 

LSC Classroom Observation Incentives
Horizon Research, Inc.
326 Cloister Ct.
Chapel Hill, NC  27514 

2002-2003 Data Collection Preparation

Data Collection Schedules

HRI will be communicating with PIs in the next few weeks to confirm LSC project data collection schedules, based on the project end dates.   Project-specific schedules for the 2002-2003 data collection year (reflecting any "data collection penalties" imposed by NSF) will be available in the Lead Evaluator and PI sections of the website after the October 15 deadline for protocol submission for the 2001-2002 data collection year.

2002-2003 Data Collection Manuals

The electronic version of the 2002-2003 Data Collection Manual was posted on the HRI LSC website on September 3, 2002.  Hard copies of the manual have been mailed out to the Lead Evaluator and PI of each project.  Projects may request additional copies from HRI. 

Sampling Frames

Updated school and teacher sampling frames for the 2002-2003 Data Collection Year are due to HRI by December 2, 2002.  To facilitate this process, HRI has recently provided preformatted sampling frame files, containing data from previous years, for each project to download from the project-specific PI or Lead Evaluator section of the HRI LSC website.  Please note that while we are making the files available in both the PI and Lead Evaluator section of the website, the project should submit only one school and teacher sampling frame to HRI.  Projects should download the files, update the appropriate information and add any additional schools/targeted teachers, and return the updated files to HRI.  Because HRI uses these files to draw samples for core evaluation data collection activities, it is important that these files be as accurate as possible, with correct treatment hours for teachers.  Consult the 2002-2003 School and Teacher Sampling Frame Guidelines, under Tab 3 of the 2002-2003 Data Collection Manual for the correct procedures for updating the files.  These guidelines can also be accessed on the HRI LSC website via the PI/Lead Evaluator sections or the online manual. 

As in previous years, Sampling Frames will be "cleaned" and samples drawn in the order that they are received at HRI.  Projects are encouraged to submit their Sampling Frames as soon as possible in order to begin Data Collection in early February. 

Data Collection Reminder

When adding individuals to the evaluation team, please note the following:

• NSF must approve all members of the evaluation team.  Vitaes of these individuals should be sent to the project’s program officer with a note describing the person’s proposed role in the evaluation;
• The Lead Evaluator must personally conduct at least three of the required professional development observations each year; and
• Individuals conducting classroom and professional development observations must successfully complete HRI’s certification/training process.

Please notify HRI of any Lead Evaluator/PI assignment changes that may occur during the year. 

Project Team Interviews

Lead Evaluators are reminded that interviews with project staff should be conducted early in the data collection year to obtain information regarding project plans that will help in selecting professional development sessions to be observed.  An interview protocol is available to assist evaluators in collecting this information; it can be found in Tab 8 of the 2002-2003 Core Evaluation Manual.

Additional project team interview protocols are provided in Tab 8 to assist evaluators in collecting information that will contribute to answering the core evaluation questions.  Consult the Project Team Interview Guidelines [PDF] or the HRI LSC website for your project’s data collection schedule (after October 15) to determine which protocols to use this year.  Please note that these interviews are for Lead Evaluator use only and should not be sent to HRI. 

Supplementing the Core

Next month HRI will be contacting projects to find out if they want to supplement the core evaluation data collection activities by administering "extra" teacher questionnaires.  Projects that are scheduled to administer teacher questionnaires in the spring of 2003 as part of the core evaluation (projects in their Baseline Year, Year Two or Final Year of data collection) may request extras if they wish to survey teachers in addition to the random sample selected by HRI, whether that is the rest of the teachers in the project or some particular subgroup.  Projects not scheduled to administer teacher questionnaires in 2002-2003 may choose to request them for all of the teachers in the project, a random sample, or a particular subgroup.  HRI anticipates being able to provide extra questionnaires to all projects as long as they are requested by November 1, 2002.  Please note that HRI will provide questionnaires and scanning as part of supplementing the core evaluation, but will analyze and prepare tables only for those projects administering questionnaires to the population of teachers or randowm sample of 300 drawn by HRI.

Meeting Plans for 2002-2003

The annual Lead Evaluator meeting will be held on January 13 - 14, 2003 in Chapel Hill, NC.  Tentatively, the meeting will begin with dinner and an evening session on Monday, and conclude with an 8:30am-3:30pm session on Tuesday.  Please note that attendance at this meeting is at project expense, and is required by NSF.

The Classroom Observation training is scheduled for February 6 - 7, 2003 in Chapel Hill, NC.   All classroom observers must successfully complete HRI’s certification process in order to be eligible to conduct classroom observations for the core evaluation.  NSF has indicated that all observers must have the necessary content background and science/mathematics education experience to conduct classroom observations.  As with all members of the evaluation team, all observers must be approved by NSF; potential observers must be approved by NSF prior to attending the training. 

Specific information regarding these meetings, including details on registration, will be sent out to all projects within the next few months.

New LSC Contact at HRI

Kristen Malzahn has taken over the position of LSC contact person at HRI from Elizabeth Arnold who has returned to graduate school.  Kristen recently left the classroom and has been at HRI for the last year as a research associate working on a number of evaluation projects.   In the past few months, she has worked with Elizabeth and other HRI staff to learn about the LSC.   The e-mail address remains the same (lsc@horizon-research.com) if you need to contact HRI for LSC-related questions.

Annual LSC Core Evaluation Requirements

All Projects:

• Professional Development Observations (5-8 per year; 2-5 for Baseline)
• Principal Questionnaires (population)
• Individual Teacher Interviews (10; not conducted during Baseline Year)
• Project Team Interviews
• Project Strategies Form (not required for Baseline Year)
• Core Evaluation Report

In addition, Baseline Year, Year Two, and Final Year Projects:

• Classroom Observations (16 per subject; 24 for multi-subject projects)
• Teacher Questionnaires (300 per subject, plus program sample of up to 75 teachers)

Most data collection activities are conducted February-May annually, except Professional Development Observations and the Project Team Interviews, which may be conducted throughout the data collection year (September 1-August 31)

The PI is responsible for making sure that core evaluation data are collected in a timely fashion and reported to HRI each fall for inclusion in the annual cross-site report to NSF.