CPS Evaluation - University
Monday April 01, 2002:
| CLASSROOM RESPONSE SYSTEM: AN EVALUATION AT AN EASY-ACCESS REGIONAL UNIVERSITY
|
by
Michael D. Everett, PhD, Department of Economics, East Tennessee State
University, and Richard A. Ranker, EdD, Office of Information Technology,
East Tennessee State University
|
| |
| INTRODUCTION |
| Classroom Response
Systems (CRSs) are a class of instructional tools that elicit student interaction
by collecting and displaying student responses to questions. This paper
describes one CRS, its equipment requirements, typical use and costs. The
interaction provided by this CRS is studied, using students in introductory
economics classes at an easy-access regional university. The time required
to learn and use the system is reported. Non-CRS- and CRS-using student
pre- and post-test performance on a standardized test are compared. CRS
student satisfaction data from an eight-question survey are presented. Finally,
reflections on the impact the CRS has had on the researcher's teaching are
presented. |
| |
| DESCRIPTION |
| Classroom Response
Systems are one way of encouraging student interactions. The CRS computer
software allows the instructor to write questions and then project them
on a large screen to the class. Students signal their answers via a small
handheld infrared transmitter. After the instructor calls time and ends
the input period, the computer instantly tabulates the answers and gives
a read out on the screen of the number of students selecting each choice.
The instructor can give immediate correction if enough students missed the
right answer. In an alternate application of the CRS, the instructor can
print out a series of questions on paper and give them to the students to
take at their own pace by clicking an answer for each question. Then the
instructor can call up a report which summarizes the class performance in
a number of ways. For example, the instructor can project an item analysis,
which shows the percent of students who choose each possible answer (a,
b, c, d, e) on a question, and give in-class feedback on the questions which
large numbers of students missed. |
| |
| CRSs have been in use
since at least 1976. The first were permanently-mounted (hard-wired) systems;
these were followed by portable systems. For example, the Q-System 1
by Reactive Systems, Inc. (RSi), was introduced in the mid-1980s. These
systems of up to 128 response pads could be connected using standard telephone
wire and RJ-11 connectors through an interface box to the computer via the
RS-232 serial port of an IBM PC or PC Junior. Similarly, the Classtalk by
Better Education, Inc., was introduced in 1985 2 |
| |
| Many have seen a CRS
in use on the television show, "Who Wants to Be a Millionaire?"; it is the
TV-remote sized tool used to pool the audience. There are several manufacturers
of CRSs. They include Classroom Performance System (CPS) from eInstruction,
Inc., the Personal Response System (PRS) from Better Education, Inc., the
Audience Participation System from Reactive Systems, Inc, and the TI-Navigator
from Texas Instruments, Inc. The CRS described and used in this research
is the CPS. |
| |
| Components:
The CPS (Figure 1, below) consists of an infrared transmitter (or pad),
a receiver, and software that collects the individual student responses
to instructor-posed questions and displays the results. One battery-powered
infrared transmitter or pad is normally given to each student. Typically
sold in sets of sixteen to thirty-two, these transmitters have eight buttons
on them and must generally be pointed in the direction of the receiver as
the students' responses are sent. Each transmission contains both the student's
response and the unique number of the transmitter or pad. The optimal range
of a pad is 35 to 40 feet, but can function upwards of 80 feet. Currently
the CPS handles up to a maximum of 256 pads. |
| |
 |
| Figure 1: The components of the Classroom Performance System (CPS) showing five transmitters, the receiver (top left) and the carrying bag with software. |
| |
| The receiver converts
the infrared signal from each transmitter to a digital signal which it sends,
via a normal RS-232 cable, to the computer software. The software collects
the digital signals and stores the student response in the location designated
for that transmitter number. It collects all responses and displays which
transmitter numbers have responded to that question (Figure 2). When the
instructor has all responses, he or she can advance to the next question.
Results can be displayed numerically, or as a histograph (a horizontal bar
graph, vertical bar graph or pie chart.) Student performance data can be
exported in a variety of formats including .doc, .rtf, .xls, .pdf or the
proprietary grade book file (.gbf). |
| |
 |
| Figure 2: Example of a screen generated by CPS. At the bottom of the screen, the grid of numbers indicates student pads that have responded in dark, pads that have not responded in white, and pads not in use in gray (25
- 32).
|
| |
| Equipment requirements:
The software was made to run on a PC in a MS Windows environment. A
Macintosh version was just released in January 2002. The PC system requirements
are very low - Windows 95 or better. However, the CPS requires a computer
to drive the software and a projector is recommended. |
| |
| Typical use: According
to the CPS website at http://www.eInstruction.com/highered.htm,
the CPS can be used to "take attendance; give and grade objective pop quizzes;
stimulate class discussion with subjective and objective questions using
CPS's ad hoc or formal question authoring capabilities built within CPS;
give a formal paper-based class test with multiple versions of the test
using CPS to grade the exam; use the CPS Gradebook to manage all aspects
of your students' grades while leveraging CPS to collect and record them
instantly." |
| |
|
At http://www.eInstruction.com/learnaboutcps.htm,
the uses of the CPS are listed and include:
- "Streamline your
grading. Your CPS results can be easily exported to Excel, Word, PDF,
or the CPS Gradebook.
- Pass out a paper
test and allow students to answer at their own pace. CPS' automated
assessment feature lets students answer test questions at their own
pace while keeping track of answers and grades behind the scenes.
- Use CPS to increase
benchmark test scores. CPS Online gives you immediate results school-
and district-wide - making CPS an ideal tool for benchmark testing.
- Provide a non-threatening
environment allowing all students to participate - even the shy ones."
|
| Cost: According
to the CPS site at http://www.eInstruction.com/cpspricing.htm,
the cost of a complete CPS is "$1,195-$3,995 depending on the size of classroom
from 16 to 32 students. Pricing for larger classes, up to 256 members, is
also available." However, this does not include the cost of the computer
required to drive the software or of the (optional but recommended) projector
to display the results. Additional equipment purchased at ETSU includes
a carrying bag for $30 and an extra receiver for $250. |
| |
| INTERACTION |
| The CPS allows students
to respond to multiple quizzes, test and surveys in class. This can increase
students' active participation in their learning, particularly in institutions
with passive learning cultures where there is typically little student time
spent outside of class on homework and course preparation. If the active
learning model -- demonstration followed by extensive student practice with
corrective feedback (Chickering and Gamson 3 , and Everett and
Zinser 4 ) - is correct, then CRS should increase student learning.
|
| |
| Through the use of
quizzes on homework topics at the beginning of class, frequent feedback
surveys at the end of each lecture segment, and discussions based on student
response patterns, students should be more actively involved in their learning.
This becomes important in easy-access schools with low standards and students
who will not spend much time studying on their own outside of class. CPS
use should result in measurable learning gains above traditional passive
lecture classes. |
| |
| Active learning is
most often the result of "interactive teaching" according to Abrahamson
5 . He expands on the options that CRSs provide, and suggest
using a variety of approaches. These provide the ability to quickly check
on both the student understanding of the content of a presentation and the
accuracy of student notes right after a lecture. The above benefits should
increase student satisfaction with the course. Structure, clarity and enthusiasm
have been the major determinants of student evaluation scores. If the instructional
use of the CPS increases clarity, student evaluations should go up. On the
other hand, passive lectures and recognition of pre-digested bits of information
on objective exams (paraphrased from Richardson, et. al., cited in Everett
and Zinser 4) constitutes the educational culture in many easy-access
regional schools. To move from this culture to active learning risks creating
confusion, frustrations and lower student evaluations in such institutions.
|
| |
| An example of the frustration
that can result from an interactive teaching approach is provided by Dufresne,
et al 6 . A CRS was being used at the Harvard Business School
by Professor Elon Kohlberg teaching managerial economics. Kohlberg pointed
out that the CRS-charted histogram was evenly split three ways. In the ensuing
discussion, he sought different students to defend their particular choices.
Then he invited the class to change their answers as he conducted another
survey. The resulting histogram was split two ways now. An impatient student
asked, "You can see we don't know where we are going with this. Why don't
you just tell us what you want 7?" Although Kohlberg was able
to quell this student's obvious frustration and engage him in the presentation
of an argument that later produced unanimity, it is important to note that
not all students find it easy to be interactive learners. |
| |
| A CRS can involve the
entire class in discussions of issues, instead of being limited to just
a few, if any, willing discussants. For example, the instructor or a student
can present an issue, then class members can discuss it in a full class
or small group setting, and then everyone can respond. Because of the anonymity
in CRSs, even the shy students can interact without cause for concern. This
too should increase student satisfaction as metered in end of semester student
evaluations of faculty. |
| |
| The type and focus
of questions are critical to effective use of CRSs. Abrahamson7 stresses
the importance of questioning style in his piece on interactive teaching.
His conclusion includes the statement that "Good questions asked in the
right context have a remarkable property to transfer a classroom." A study
of IBM management trainees at the IBM Corporate Education Center 8
using the Q-System CRS from RSi also showed that questioning style was the
key. In this study, researchers recognized five types of questions (true/false,
multiple choice, mean numeric entry, correct numeric entry and rating scale
1 to N) as well as several interesting variations of those five. After "relevance
and quality of the student response questions were enhanced and soliciting
student responses more timely and frequent, i.e., at least once every 15
to 20 minutes, " attentiveness and retention were improved. "Thought provoking
questions stimulated the students' desire to seek self-discovery and provided
them with the opportunity to compare their answers to those of the rest
of the class and their peer groups. The subsequent discussions explored
issues at greater depths and encouraged participation from a much larger
percentage of the students. Classroom time was used more productively and
[the materials could] be covered in the same allocated time. The technology
assisted the facilitator to complete all the learning points within the
allotted time and in some cases the total number of learning points were
increased by 20 to 30 per cent." The researchers of this paper sought to
determine if the use of the CPS would have a similar effect on student learning,
as measured in a standardized test of comprehension, or student satisfaction,
as measured in end of course student ratings of faculty and a short survey
of satisfaction with the CPS system. |
| |
| PERFORMANCE |
| The researcher administered
questions 1 through 30 of the Test of Understanding College Economics, which
had a national norming group of about 2,700 students from Ivy League to
Junior College. Students in six different classes of microeconomics were
administered the same test at the beginning and at the end of the course.
Performance data are presented in Table 1. |
| |
 |
| Table
1: Scores of Everett's microeconomics student on the Test of Understanding
College Economics.
|
| |
| Data were grouped by
treatment; the Fall 2000 and Spring 2001 students did not use the CPS system
and thus their test scores were grouped together. Similarly, the Summer
2001 and Fall 2001 students did use the CPS and thus their test scores were
grouped together. Although the total number of students involved in these
treatments were small enough (60 in the non-CPS group and 56 in the CPS
group) to make statistical comparisons questionable, some comments are appropriate.
First, the averages for both the non-CPS and CPS groups were slightly lower
(32.2 and 31.3) than the national average (35.7) on the pretest. Second,
the averages for both the non-CPS and CPS groups were slightly higher (52.9
and 53) than the national average (51.2) on the post-test. However, the
researcher had used an aggressive active learning approach in the non-CRS
courses, using much in-class and homework drill. Third, since the CPS group
started slightly lower and ended up slightly higher than the non-CPS group,
it earned a slightly higher gain (21.7 percent or 69.2 percent gain) than
the non-CPS group (20.7 percent or 69.2 percent gain). Fourth, the gains
for each class in the CPS group are very consistent - within 0.4 percent
of each other - while the gains for the three classes in the non-CPS group
vary by 10.6 percent. In summary, the pre-post comparison of the non-CPS
and CPS students on the TUCE, while not demonstrative, are encouraging and
warrant continued study. This is especially true since the professor prided
himself on the fact that prior to use of the CPS, he considered himself
to be very interactive in his treatment of the students. |
| |
|
STUDENT
SATISFACTION SURVEYS
|
| There were two separate
tests of student satisfaction. The first was the plan to compare end-of-semester
student ratings of faculty; the second was a plan to survey students on
their satisfaction with the CPS. Details of both follow. |
| |
| Student Ratings
of Faculty: Students have only formally evaluated two of the three CPS
classes. Those evaluations were in the same range of average evaluations
normally obtained in microeconomics classes by this professor. Because of
the low numbers of evaluations completed, no certain generalizations could
be presented. However, it appears that the one class found the presentations
very clear, a major determinant of relatively high evaluations, and the
other class found the presentations unclear. Thus, the ability to use the
CPS to quickly survey the students' understanding of basic points and clarify
points which students missed after a short lecture, apparently may not have
increased the overall clarity. The low number of participants in these evaluations
raises serious reliability issue. Perhaps further evaluations (more years)
will show a more positive trend. However, the professor's rigorous active
learning approach in a passive lecture environment may continue to negatively
affect evaluations. |
| |
| Survey of Student
Satisfaction with CPS: On the affective (feeling) side, students seemed
to like the CRS. On written surveys administered to four separate classes,
60 students indicated a high level of satisfaction with the CPS [Table 2].
|
| |
| The surveys consisted
of eight statements listed below. Students were asked to respond to each
statement using: a. Strongly Agree, b. Agree, c. Disagree, d. Strongly Disagree
and e. No Opinion. Responses were converted to ordinal data by equating
a Strongly Agree response to 1, Agree to 2, Disagree to 3, Strongly Disagree
to 4 and not including No Opinion responses. Average responses for each
of the eight statements are presented in Table 2. |
| |
|
For purposes of interpretation,
any statement with an average response of:
- 1.0 to 1.5 was
considered to be in the "Strongly Agree" range;
- 1.51 to 2.5 was
considered to be the "Agree" range;
- 2.51 to 3.5 was
considered to be in the "Disagree" range; and
- 3.51 to 4.0 was
considered to be in the "Strongly Disagree" range.
|
|
The eight survey statements
on the CPS (called the Student Class Response System or SCRS in the survey),
the average numerical response each was given by the students, and their
interpretation were:
- The SCRS helped
me understand the lectures better by catching and correcting my mistakes.
- 1.53 - Agree
- Using the SCRS
after a short lecture took time that could have been more productively
devoted to lecture. - 3.20 - Disagree
- We really needed
to write the basic concepts out first before trying to answer objective
questions on them. - 1.90 - Agree
- The SCRS helped
increase learning outputs by involving students in more active learning.
- 1.53 - Agree
- Learning takes
place best when instructors clearly provide information to students
rather than requiring students to write, think about, and answer questions
on information given them. - 2.99 - Disagree
- I enjoyed the increased
class involvement the SCRS provided me. - 1.68 - Agree
- The SCRS system
increased my level of anxiety and tension since I had to participate
whether I wanted to or not. - 3.20 - Disagree
- I do not like active
learning and thus do not like the SCRS. - 3.51 - Strongly Disagree
|
|
|
Table
2: Results of survey of 60 students enrolled in microeconomics classes
in which the CPS was used.
|
| |
| In summary, the students
either disagreed or strongly disagreed with all statements that denigrated
active learning and/or the use of the CPS. The students agreed with all
statements that spoke well of the CPS (SCRS) use. The CPS was a big hit
with the students. |
| |
| TIME DEMANDS |
| In addition to the
modest monetary costs, the researcher invested substantial but manageable
time costs. Most of these costs involved time which could have been devoted
to other teaching, research, or service activities. For example, about 50
hours were spent in first-time set-up costs learning the program (mainly
on his own), writing the questions before each class, and arranging for
assignment of classes to multimedia classrooms (see http://ats.etsu.edu/mc/)
to assure availability of the equipment needed to use the CPS. Now that
the initial problems associated with initial set-up have been resolved,
the researcher or the Office of Information Technology staff could probably
teach the program to other faculty in a two- to three-hour workshop. Most
of the other set-up time would also be significantly diminished. |
| |
|
|
Table
3: Researcher's estimate of time spent on various tasks associated
with CPS use. Note that First Time Overall Set-Up Costs and Overall Initial
Class Set-Up Time are reported in hours; Per class Additional Time Costs
are reported in minutes.
|
| |
| Some time costs associated
with CPS use will not be diminished. The researcher estimates between 5
and 25 minutes additional time per class in setting up and using the system
for that class; the total could reach up to 40 minutes, if questions need
modified. Table 3 describes the tasks associated with CPS use, provides
rough estimates of the minimum and maximum time the researcher spends completing
those tasks, and provides a format for more systematically collecting time
and motion type cost data. Time demands, while formidable in the first-time
set up period, are much more manageable for the next faculty users. |
| |
| DISCUSSION OF IMPACT ON TEACHING |
| This study has attempted
to identify and measure a wide range of benefits and costs associated with
using the CPS in introductory economics classes at an easy access regional
university typical of many regional universities, colleges, and junior colleges.
Time costs, such as learning the program, preparing questions, accessing
technical equipment, and daily loading and unloading class data can be substantial.
However, the next faculty to use this system will face a much easier path
since the initial training package has been developed and the technology
has been tested and integrated into the technology classrooms on the campus.
The remaining time costs may be counterbalanced by other benefits, such
as student interest and attention and a feeling of connecting with a normally
passive class. |
| |
| The following anecdotal
evidence comes from the researcher's experience teaching three, and half
way through two more, introductory economics classes with the Classroom
Performance System. Table 1 shows the class scores on national standardized
exams remained about the same as other recent classes the researcher has
taught: relatively low pre test scores and relative high post test scores
and gains, compared to national averages. However, it does not reflect the
observation that these classes seemed to cover the basics more quickly and
thus had more time for broader discussions. In previous (non-CPS) classes,
the researcher would give short lectures and then assign in-class exercises
and practice quizzes. Time would be spent walking around the classroom checking
students progress and/or have other students provide feedback on each other's
work. This took more time than the CPS and provided much less reliable information
on problem areas experienced by the students. On the other hand, the CPS,
with its objective question formats, emphasizes recognition of the "right"
answer rather than working through the graphical analysis. This points to
the need to improve the question drafting process to use more graphically
based questions. |
| |
| The CPS seemed to leave
more time for broad analyses of articles and cases in the textbook. It improved
these discussions by letting all students vote on policy issues which had
been debated in class. This gave the researcher a feeling of connection
with classes of students who traditionally are very passive, uninvolved,
and often bored with virtually any kind of academic material |
| |
| These studies have
not yet yielded data on the CPS's ability to motivate real homework study
as opposed to just going through the motions. In the initial design, the
CPS was to be used to quiz students on the homework concepts, thereby motivating
students to complete all assignments prior to coming to class and to actually
learn the principles through practice. However, transferring the data from
the CPS Gradebook program to the class' official grade spreadsheet remains
time consuming (at least 5 minutes per assignment) and did not get done.
An update of the current version of the CPS software may result in a more
efficient grade book. Also, commercial Internet testing services (e.g.,
http://www.Gradesummit.com)
may perform this function more efficiently by allowing instructors to assign
an exam over topics in a textbook to test students' knowledge of that topic.
Again, the adoption of this technology will depend on the ease-of-use of
its record keeping formats. |
| |
| These researchers believe
the CPS and other CRSs can efficiently improve active learning and learning
outputs in easy access schools where most students spend very little time
studying outside of class. Surveys indicate students like the CPS. Pre-post
standardized testing hints at the possibility of increased learning due
to the CPS use, but more study is required. Perhaps the lack of significant
comparative evidence of gains on the TUCE may stem mainly from the use of
a very rigorous active learning approach before the CPS was introduced.
Class sizes of 20 to 30 students allowed this. The apparent lack of impact
of the CPS on student evaluations may be caused by the frustration students
feel when their overall passive lecture system - which most students have
experienced and seem to generally prefer - is disturbed by the very active,
mastery learning approach. However, two sets of student evaluations are
too few to draw a valid conclusion. |
| |
| There is a shortage
of hard evidence that CPS use increases learning outputs or student evaluations
in our introductory economics classes, even though the evidence is positive
and hopeful. The major intuitive benefit for the researchers is a feeling
of connecting with classes of traditionally very passive students. This
comes both from feedback on basic lecture material and class discussions
on broader issues which the CPS system facilitates. |
| |
| CONCLUSION |
| In conclusion, the
theoretical and intuitive benefits seem to outweigh the preparation and
set up costs. CPS use appears to increase active learning by devoting some
classroom time to student involvement in active feedback such as quizzes
on lectures. The theoretical and empirical literature on active learning,
cited above, seems compelling. These observations hold particularly true
in easy-access regional schools where students do very little work outside
of class. On the other hand, such schools tend to emphasize short-term student
satisfaction rather than actual learning. This constitutes a major barrier
to the use of active learning technologies like the CPS. |
| |
| Nonetheless, the pre-post
TUCE results did appear to favor the CPS group, the end of course faculty
evaluation data were mixed and too few to allow generalization, the student
endorsement of CPS use was strong, and the time studies indicate that the
time commitment for other faculty to use the CPS will have been significantly
diminished. The major intuitive benefit for the researchers is a feeling
of connecting with classes of traditionally very passive students and the
time it makes available for other higher-order thinking. Further study seems
to be in order. The possibility of further study will be enhanced by the
recent decision to place the CPS in the remaining multimedia classrooms
at ETSU and make it part of the standard package for new classroom installations.
|
| |
| REFERENCES |
| 1 Q System
Class Product advertising sheet, Reactive Systems, Inc., Englewood, NJ |
| |
| 2 Better
Education brief history, http://www.bedu.com/ |
| |
| 3 Arther
Chickering and A. F. Gamson, "Principles of Good Practice in Undergraduate
Education," The Wingspread Journal, special section, June 1987 and reprinted. |
| |
| 4 Michael
D. Everett and Otto Zinser, "Interdisciplinary Social Science Courses: Using
a Critical Thinking Approach," Journal of General Education, Vol. 4, No.
3. 1998 |
| |
| 5 A. Louis
Abrahamson, "What IS Interactive Teaching?", http://www.bedu.com/interactive.html |
| |
| 6 Dufresne,
R.J., Gerace, W.J., Leonard, W.J., Mastre, J.P., and Wenk, L., "Classtalk:
A Classroom communication System for Active Learning," Journal of Computing
in Higher Education, 7, 3-47, 1996 |
| |
| 7 A. Louis
Abrahamson, "An overview of Teaching and Learning Research with Classroom
Communicatins Systems (CCSs)," paper presented at the International Conference
of the Teaching of Mathematics, Village of Pythagorion, Samos, Greece, Conference
Proceedings by John Wiley & Sons, Inc., June 3-6, 1998; also available at
http://www.bedu.com/Publications/Samos.html |
| |
| 8 Harold
M. Horowitz, "Student Response Systems: Interactivity in a Classroom Environment"
Xerox Palo Alto Research Center, 1984; also available at http://www.rsicommunications.com/keypads/study.htm;
later updated at http://www.meeting-net.com/users/classroom_interactivity.pdf |
|
