Field Testing
Field testing of the CAM began on May 1st and is currently underway. The primary lines of inquiry surrounding this process include: (1) gaining insights into the CAM’s suitability for use as a summative and formative assessment tool in Ohio's school districts, (2) discovering any necessary areas for improvement in its design, and (3) identifying training and support requirements for end-users and support staff.
Requirements for Participation
To participate, districts must meet the following requirements.
District Participation
- At least one administrator or individual with appropriate authority from each district has been identified who will coordinate the district's efforts to implement the CAM.
- The district leader has agreed to attend an initial introduction to the system.
- The district leader will train teachers within the district to use the system for the purpose of administering summative and formative assessments. Alternatively, the district leader has arranged for their ITC or the D3A2 team to train the district's teachers.
Implementation Plan
- The district leader plans to involve a select group of teachers and students in field testing.
- The district has a collection of test questions that are suitable for input into the CAM and which it intends to use for its summative assessment program.
- The district has completed all necessary prerequisite activities for data loading and account set up.
Evaluation
- The district leader has agreed to participate in regular web-based meetings to share feedback about their experience using CAM with the other pilot districts.
- The district leader has agreed to complete an exit survey by June 15th to provide input about their district's experience using the system.
ITC Participation
- The district's ITC has expressed its willingness to assist during field testing by preparing necessary data files, attending web-based meetings, and provide application support as the district learns to use the CAMs features and settings.
Protocol
Field testing of the CAM is governed by the protocol outlined below.
Phase I of Field Testing
Phase I of field testing will take place from May 1, 2010 to June 30,2010. Participation is completely voluntary.
The primary objective of this phase is to get a small number of districts using the CAM to provide initial feedback about its performance and suitability as an assessment tool for Ohio school districts.
Objectives
Phase I districts will be encouraged to conduct end-to-end testing of the system, including but not limited to:
- Complete all preparatory activities.
- Create question bank items as a district administrator.
- Create course sites as a teacher.
- Create tests as a teacher.
- Take tests as student.
- Score tests as teacher.
Note: it is important that this experience progress at a pace that is comfortable for each pilot district. Therefore, it is assumed that not all districts will be able to fully test the CAM during the allotted time.
Timetable
- May 1, 2010: field testing begins.
- June 15, 2010: Phase I district end-user surveys are collected.
- June 30, 2010: Phase I evaluation report submitted to ODE.
- All other meeting and training dates will be posted on the CAM Pilot site calendar.
Phase II of Field Testing
Phase II of field testing will follow the conclusion of Phase I and extend through the summer until the start of the 2010-2011 school year. Phase I districts, as well as a number of additional districts, will be encouraged to complete their district question banks and begin formulating a plan for using the CAM as a formative assessment tool with teachers and students during the 2010-2011 school year. This plan should include selecting and training a target group of teachers and students that will use the CAM during the coming year.
Objectives
Phase II districts will be encouraged to complete the following activities:
- Finalize work on any incomplete Phase I activities.
- Plan to use the CAM to conduct benchmark assessments during the upcoming school year.
- Fill out the district question banks.
- Build out benchmark assessments to be used during the upcoming school year.
- Create student and teacher accounts (with assistance from their Information Technology Centers).
- Extract 2010 class rosters from eSIS or DASL and upload to D3A2 (with assistance from their Information Technology Centers).
- Train target teachers in the use of the CAM.
Timetable
- July 1, 2010: Phase II field testing begins.
- September 15, 2010: Phase II district end-user surveys are collected.
- September 30, 2010: Phase II evaluation report submitted to ODE.
- All other meeting and training dates will be posted on the CAM Pilot site schedule.
Phase III of Field Testing
Phase III of field testing will begin after the start of the 2010-2011 school year and extend through the end of the first half of the school year. During Phase III of testing, district leaders will implement their plans for using the CAM as an assessment system with a targeted group of teachers and students.
Objectives
Phase II districts, as well as a number of additional districts, will be encouraged to complete the following activities:
- Finalize work on any incomplete Phase I and Phase II activities.
- Create course sites and copy district authored benchmark assessments into them as necessary to support their implementation plans for the CAM.
- Have target students log in and take benchmark assessments.
- Score student responses to benchmark assessments.
- Evaluate and revise initial implementation plans as necessary based upon lessons learned during piloting.
- Make plans for continued and expanded use of the CAM.
Timetable
- October 1, 2010: Phase III field testing begins.
- December 15, 2010: Phase III district end-user surveys are collected.
December 30, 2010: Phase III evaluation report submitted to ODE.
Evaluation
The following measures will be employed to assess the overall effectiveness of the system during field testing.
Written Communications
District leaders will be asked to submit written communications regarding their progress and experience twice per month to the CAM Pilot site forum. These communications will be reviewed for information about each of the district's ease planning for and implementing the CAM. It is expected that they will also provide valuable information about the overall performance of the system and any improvements that will be necessary before wide-spread use of the system.
CA-Unicenter Entries
Requests for support that are officially logged in the CA-Unicenter support desk will be analyzed and described in the Phase I evaluation report when it is submitted to ODE.
District Surveys
District leaders will complete exit surveys to document their experiences and impressions gained by using the system. These surveys will be designed to assess the following questions:
- What is the rating for overall usability of the system for district administrators? What factors contributed to this rating?
- What is the rating for overall usability of the system for teachers? What factors contributed to this rating?
- What is the rating for overall usability of the system for students? What factors contributed to this rating?
- What level of compatibility exists for the CAM within regard to the district's plans for implementing a balanced assessment system? What factors contributed to this rating?
- Is the district interested in continued use of the CAM after field testing the system? What factors contributed to this level of interest?
- How do the features of the CAM compare with other assessment systems the district has used or evaluated?
- What are the strengths of the CAM?
- What are the necessary areas for improvement?
- What was the quality of training and support provided by the ITC and D3A2 team during field testing?
Evaluation Report
The D3A2 team will submit an evaluation report to the Ohio Department of Education at the conclusion of each phase of field testing. These reports will summarize the results of all measures utilized during piloting and address the questions described above.