INTIME Evaluation Overview
Two parallel evaluation efforts are designed to address the project effectiveness both formatively and summatively. The main categories of the formative stage of the evaluation process are as follows:
Teacher Reflective Practice
a) The teacher reflective
practice takes into account project goals, indicators, benchmarks, and measures as they relate to classroom
practice. The following categorization of evaluation data sources lends itself
to understanding the complexity of the project:
- Stages of Concern Questionnaire (SoCQ) data collected from the participating faculty. The first two sets were administered at the beginning and the end of a preparation workshop offered before implementing the INTIME project. All the subsequent sets have been administered at the beginning and the end of each semester during which INTIME was used in the methods classes taught by the participating faculty.
- Revised Syllabi and Technology Integration Action Plans collected from all participating faculty as evidence of their project-based planning of classroom instruction. The accommodations made to integrate INTIME into the methods classes taught by the participating faculty are also included in the Final Course Revision Reports that have been submitted to the project management team. An in-depth analysis of all these data reveals the fact that the INTIME experienced users, as indicated by the SoCQ positive evolution pattern, have been creative and effective in the overall use of the project materials and technology.
- INTIME WebCT postings in separate fora for participating faculty and the students enrolled in their methods classes. The rationale for using this method to assess the project effectiveness relies on facilitating reflective thinking and encouraging an exchange of ideas about the integration of technology and components of quality education.
- Technology Competency Pre- and Post-test data collected from students enrolled in the classes taught by the INTIME participating faculty. The pre-test has been administered at the beginning of the semester, prior to exposing students to the INTIME project, while the post-test has been administered at the end of the same semester.
- Phone interviews with participating faculty, based on the formative evaluation of their to-date use of the project-developed resources. The INTIME management team constantly takes into account the input from all participants and focuses on possible ways to improve certain aspects of the project. The next step of the formative evaluation effort is to encourage the INTIME faculty to reflect on the facilitators and hindrances in the employment of INTIME resources.
- PT3 Baseline Survey administered to all project participants. This survey is a 35-page formal evaluation instrument used to assess the impact of PT3 grant activities on teacher preparation programs at recipient institutions of higher education.
b) Web Resources
- Website Usability Questionnaire data collected from the participating faculty. This particular instrument was created by the INTIME management team in an attempt to capitalize on the input from participants regarding their use of the project in the college classroom.
- Website server report that breaks the overall number of hits and visits over a given period of time down to top pages visited, geographic location of visitors, search engines used, etc.
c) Project Progress
Weekly/quarterly/annually project progress reports sent to
project staff and the granting PT3 central office. All these reports offer
details concerning the status of different project-related tasks and their
upcoming deadlines, project presentations at various
local/national/international conferences, etc. These
reports are used to assess the project progress and adapt any further
developments to the needs of the participants and other users.
At the end of the third year
into the project, a professional evaluation team will examine summatively
the effective impact of new learning resources and new standards developed.
Teacher Reflective Practice
Extensive data have been
collected in relation to the movement of individual faculty toward integrating
technology into their methods courses. These changes have been charted using
the Stages of Concern Questionnaire (SoCQ) from the Concerns-Based
Adoption Model created by Hall, George, & Rutherford (1977).
[The model] assumes change to be a highly personal and lengthy process, one that affects individuals differently. The model hypothesizes two dimensions along which individuals grow as they become more familiar with and sophisticated in using innovations: Stages of Concern about the Innovation (SoC) and Levels of Use of the Innovation (LoU) (Loucks, Newlove, & Hall, 1998, p.1).
Data for each of the measures are collected and used by methods faculty involved in course revision to determine the effectiveness of their strategies to integrate technology.
The following different stages of concern in the questionnaire assess several major factors that influence the adoption process of any given instructional innovation: Awareness, informational, personal, management, consequence, collaboration, and refocusing.
The first stage is related to the degree a user is aware of the innovation. A little over half of the INTIME participating faculty who have completed at least four SoC Questionnaires have high scores on the awareness stage. A possible explanation is that while they have become comfortable with the innovation, they are currently concerned with other issues not related to the project.
The second stage of the questionnaire focuses on the innovation-centered information that the user has acquired or has yet to acquire. Most of the INTIME respondents’ scores are constantly decreasing, in the average to low region. This indicates that the faculty have acquired content-specific knowledge that enables them to implement the innovation effectively.
The third stage of the questionnaire emphasizes user’s possible personal concerns about the innovation and its consequences. While these concerns reflect uneasiness regarding the innovation, they do not necessarily indicate resistance. The vast majority of the INTIME participating faculty who have completed at least four SoC Questionnaires have fluctuating average to low scores. This indicates increasing comfort using the innovation.
The next stage is related to the degree to which managing the innovation poses problems to the user. Once again, the vast majority of the INTIME respondents display average to low scores that indicate that logistics, time, and management are not hindering issues in using the innovation.
The fifth stage of the questionnaire represents possible concerns a user may have regarding the consequences of the innovation upon students. Almost all the respondents have average to low scores. This trend indicates not only increasing comfort using the innovation, but also demonstrate the effective communication between the faculty and their students during the implementation process.
The sixth stage deals with concerns a user may have about collaborating with others on the innovation. Half of the INTIME faculty who have completed at least four SoC Questionnaires display high scores on this particular stage. The most likely interpretation is indicative of possible impediments in sharing knowledge and information about the innovation.
The seventh stage of the questionnaire focuses on the various ideas and/or suggestions that a user might have gained from using the innovation. Most of the INTIME respondents have fluctuating average to low scores. This interpretation also indicates faculty comfort in using the innovation. However, the results graphs of more than half of the respondents show they have ideas/suggestions as to how the innovation could be improved or implemented differently, thus pointing out refocusing in using the INTIME project.
graphs based on the scores for at least four SoC Questionnaires indicate
increasing familiarity with the innovation. The individual differences in
scores are related to the specifics of the contexts in which each
participating faculty uses INTIME.
These data can also be tied with the information gathered from the Website
Usability Questionnaire administered to all participating faculty. Their
increasing familiarity with the innovation is to be found in the degree to
which they find the INTIME
resources accurate content-wise, effective, and helpful in their professional
Revised Syllabi and Technology Integration Action Plans have been collected from all participating faculty as evidence of their planning process centered on INTIME. The project staff identified some of the commonalities among all of these revised syllabi. The focus of this overlap is on the goals that the faculty revisited and adapted to integrate technology into their methods classes. It should also be noted that the more positive the SoCQ evolution pattern (as demonstrated by the graphs mentioned above), the more accommodating to INTIME these revised syllabi are, as follows:
a) Students are to identify the ways in which technology can support student learning.
b) Students are to view and critique the INTIME online video vignettes.
c) Students are to engage in online chat using the INTIME WebCT.
d) Students are to take a set of pre- and post-tests designed to assess their technology competencies as pre-service teachers.
e) Instructors are to model the use of varied instructional technologies in alignment with those demonstrated in the INTIME online video vignettes that students are to watch as a class requirement.
INTIME WebCT Postings have been collected from their different fora (by institution) in an attempt to use this kind of qualitative data as a way to assess formatively the project effectiveness. Taking into account the input from participating faculty and the students enrolled in the methods classes they taught at the five universities, the INTIME staff could offer assistance in adapting the project to the various needs of the users. For instance, a team of methods faculty created their own preliminary technology assessment tool based on the INTIME Technology Competencies Test.
The students who
have been exposed to the project in the methods classes taught by the
participating faculty also had the opportunity to voice their opinions and
concerns, if the case, related to the use of technology in the classroom, as
demonstrated by the INTIME
video vignettes. In addition to meeting the requirements of their class by
critiquing the video vignettes, these students also share their insights by
relating the topic to their experiences either as students or student
teachers. The following examples show the wealth of qualitative data the
project staff have had to incorporate into their evaluation efforts:
I found that the video
"Giving 'Em the Business" presented a new and entertaining way to
incorporate mathematics into the elementary school classroom. Children of this age (or any, for that matter) have a much
easier time grasping concepts presented to them through real world scenarios
rather then just learning off of a chalkboard.
The video seemed to show the students showing genuine enthusiasm for
the projects as opposed to barely staying awake, which is a common occurrence
in many math classrooms. On top
of being a great educational opportunity for math, this project also
incorporates many other areas such as technology and language arts, two very
important fields of study for the student of today.
Overall, this was a very
effective, hands-on approach to learning, which involved the kids
"doing" instead of just watching.
Plus, planning a business, marketing the product, guessing sell prices,
estimating income, is all part of the "real world", so this was a
very practical problem. Even though it was a math lesson, it incorporated many
other ideas as well. Technology
planned a vital role as well. Not
only were they using a variety of software programs on the computer to design
pictures for their business and calculating profit, loss, spending, etc. they
also were incorporating other technology, such as the digital camera. It was
used effectively, but was not the only thing they used. They worked technology into the project, but did not over use
it, which I thought was important.
I agree that it was important
that technology was incorporated, but not overused.
You want for your students to have a solid background in technology but
you do not want it to overshadow the basic principles that you are trying to
I agree with what you said. I
had the same feelings about technology as you did.
I liked how you also said that they used many different software
programs. I had forgotten about them when I wrote my response. The things
presented in this lesson do allow the students to feel like they are apart of
the "real world." I had
commented that they felt like adults.
We have also been collecting data on the evolution of technology competencies for students enrolled in the classes taught by the INTIME participating faculty prior and after implementing the project. While implementing the new online learning resources and standards, methods faculty could model the use of instructional technology to their students. Therefore, the technology competencies of pre-service teachers would change over time either positively or negatively. These competencies have been evaluated formally by administering a set of Technology Competencies pre- and post-tests at the beginning and toward the end of a semester when the INTIME project would be used in a methods class.
A negative change on the post-test compared to the pre-test would point out the fact that students could have over-rated their technological skills before using the INTIME project in class. Under these circumstances, the lower post-test scores would be interpreted as students’ better self-reflection regarding their skillfulness in using technology in the classroom as demonstrated by the INTIME video case studies they analyzed in class.
A positive change
could rely on the learning curve that students go through while using the INTIME
project in class, as well as on the methods faculty’s modeling of
appropriate uses of instructional technology.
No change in
technology competencies could indicate various hindrances that affected the
learning process assisted by the INTIME
project. In this case, the methods faculty’s insights on the dynamics of the
class would reveal the possible causes for the zero difference between the
technology competencies pre- and post-test.
For instance, the data collected
for the Fall 2001 semester during which the INTIME
project was implemented in various methods classes at all five participating
universities revealed no change in technology competencies. 269 students took
both the pre- and post-tests. A little over half of those students showed no
change in their technology competencies. An accurate analysis of the figures
would have to take into account the specifics of each class using the INTIME
Baseline Survey represents yet another set of data that helps in assessing
the effectiveness of the INTIME
project as a PT3 grant. An initial look at what a PT3 grant represents in
today’s communication era would put things into the right perspective:
Federal, state and local
agencies are investing billions to equip schools with computers and modern
communication networks, but only one-third of our nation's teachers feel well
prepared to use computers and the Internet in their teaching.1
With 98 percent of our schools
connected to the Internet and the powerful learning that the Internet makes
possible, profound changes in the way future teachers are taught are necessary
if we are to meet the demand for teachers prepared to educate 21st century
In response to this need, the
Preparing Tomorrow's Teachers to Use Technology (PT3) initiative awards grants
to consortium partners that are working to transform teacher preparation
The challenge of transformation
is daunting. PT3 grantees are developing models, tools, support and incentives
to help faculty make the change to technology-infused teaching, both within
schools of education and throughout the campus. (available at www.pt3.org)
Under these circumstances, the impact of the INTIME project on the five participating colleges of education is of substantial importance in assessing the effectiveness of this particular PT3 grant. Based on the data collected from our INTIME faculty, some major differences occurred between the period before and after project implementation:
In addition, faculty have been using more technology in instruction in the following areas: technology basics, using the Internet to provide essential course information to students, using multiple media to communicate information in the classroom, and requiring students to develop lessons and make in-class presentation using technology.
Faculty also offered more core
courses in which students are instructed on basic technology skills, classroom
information management, ethical issues related to technology, and ways to
integrate technology into the curriculum. As a result, faculty perceive their
students as more proficient in terms of technology operation and concepts,
planning and designing technology-enhanced learning environments and
experiences, teaching, learning, and curriculum, assessment and evaluation,
and productivity and professional practice.
The Website Usability
Questionnaire is an instrument designed by INTIME
staff members to evaluate usability of online resources developed by the
project and administered among faculty-participants demonstrates that for the
vast majority of respondents the overall resources were (1) easy to use, (2)
useful for instruction, (3) supportive of teaching and learning styles, and
(4) content accurate and free of errors.
When asked to comment on each of the resource components, the
faculty-participants expressed overall satisfaction with the format and
content of video examples and teacher insights available on the site.
In regards to the theoretical framework of the project – TFQE
Model – the faculty also found it to be useful and easy to use for them
and their students. In spite of
its later addition to the site, the Probing Questions feature became quite
popular among faculty-participants, according to their responses to the
instrument. The majority of them
started to use it as soon as it became available and reported that it was
user-friendly and beneficial. Based
on the data collected using this questionnaire, the majority of faculty
participants used project developed online resources quite frequently in their
classes and found them to be supportive of teaching and learning. One example
along these lines is offered by the comment one participant makes with regard
to the overall usefulness of the INTIME
This Web site is helpful to
visual, auditory, and tactile learning. Students like to see the videos and
read the lesson plans, articles, and video captioning. Others like the
auditory features of the videos. Tactile learners like the computer aspect.
Summary: TELL ME and I’ll forget. SHOW ME and I may remember. INVOLVE ME and
I’ll understand. This Web site does all three.
Server Reports data collected using Webtrends Inc., software is another way to assess the impact of the project and its resources on a regular basis. These reports provide specific information on visits to the INTIME site such as: who, from where, how often, for how long, pages most frequently hit, etc. The following statistics reflect the most recent evolution in the life of the Web site: a) number of successful hits to date: 4,124,125; b) number of visits to date: 114, 964; c) average number of hits per day: 5,412; d) top geographic region: North America (82,963 visits); e) most active countries: US (81,873), UK (1,136), and Australia (1,071).
project progress reports sent to project staff and the granting PT3
central office provide stakeholders with specific information related to the
various tasks within the project, past and upcoming formal presentations of INTIME
at different local/national/international conferences and workshops, as well
as the status of the online resources production. Therefore, the granting
organization as well the project personnel would be able at any given time to
assess how the project goals and objectives are constantly being met. In
addition, these reports also provide information related to new directions in
which the project is going.
The instruments of both formal
and informal evaluation show that the INTIME
project has had a major positive impact on the participating faculty and their
students. The long-term effects of the project on the teacher education
programs involved will be displayed once these students go out into real
classrooms and apply their knowledge and skills. Therefore, the INTIME
project represents an invaluable resource for all those interested in
improving our schools by preparing better teachers and it may serve as a model
for future similar projects.
& Switzer, T.J. (2001).
Technology as facilitator of quality education:
A model. In W.Heineke
& J. Willis (Eds.), Methods of evaluating educational technology
(pp. 215-235). Greenwich, CT: Information Age Publishing, Inc.
Hall, G. E.,
George, A. A., & Rutherford, W. L. (1998).
Measuring stages of concern about the innovation: A manual for the
use of the SoC Questionnaire. Austin,
TX: Southwest Educational
Society for Technology in Education (ISTE).
educational technology standards for teachers. [Online] Available: http://cnets.iste.org/index3.html
[2002, March 15]
Hansen, L., Smaldino, S. (2000, April). Preservice teacher technology
competencies. Tech Trends
44 (3) 47-50.
Copyright. © 1999-2001, INTIME.
Privacy Statement Credits