The Efffect of Question's Types and Levels on Students' Academic Achievement
 
Home
Author
Supervision
Abstract
Chapters
References
Appendices
Contact

Chapter Three
 
Population Sample Experimental design
Instructional task Experiment and control groups Test reliability
Procedures

Scoring

Measures Statistical design

Population
The population of this study consisted of all students who were enrolled in "University English I" course (10103), the first semester 1996 / 1997 at An-Najah National University, Nablus, Palestine. The whole population was (935) students, most of them were at a freshman level.

Sample

A random sample for this study was taken from the whole population; it consisted of (140) students who came from different colleges at An-Najah National University, Nablus. Almost one third of the sample was from the Faculty of Engineering, the rest of it came from different faculties at the An-Najah. Students ages ranged from 19- 21 years, with an average of 20. See, Table ( 1 ) . 

Table (1) shows the number of students who participated in the study and the name of the college which students are affiliated to.

Name of the college   Number of students
Faculty of Arts                    26
Faculty of Science                   13
Faculty of Economics                  32
Faculty of Education                    7
Faculty of Engineering                  55
Faculty of Islamic Law                    2
Faculty of Fine Arts                   1
Faculty of Pharmacy                   2
Faculty of Law                   2
TOTAL                140

Experimental Design TOP

The experimental design was the posttest - Only Control Group Design. The diagram for this design appears in the following figure:
                                                R         X         O 
                                                R                     O

= = = = = = = = = = = == = = = = = = = = = = = = = = = = = = = = = 
R = Random assignment 
X = Independent variables 
O = Observation or test

Statistical Design TOP
= = = = = = = = = = =

A Three - Way Analysis of Variance (ANOVA) factorial design (2 x 3x3) was used for analyzing the data of the study by using general F-test. The first factor was the type of questions (essay versus multiple-choice questions), the second factor was the level of questions ( RI, RG, and UG), and the third factor was students’ abilities (high, medium, and low). See, Table ( 2 ).

Table( 2 ) shows the three variables of the study: question types, question levels, and student’s ability.
 
 
Question types Multiple-choice  Essay
Question levels 
RI
RG
UG
RI
RG
UG
High student’s ability 
Medium student's ability
Low student's ability

One Way Analysis of Variance will also be used ( 1x7 ) to compare the experimental groups with the control group (See, Table 3 ).

Table ( 3 ) shows the experimental groups and the control group which were used in the study.

RIM 
RGM 
UGM 
RIE 
RGE 
UGE 
Control 

RIM   =   Remember- an - Instance Multiple-choice. 
RGM  =  Remember - a - Generality Multiple-choice. 
UGM  =  Use - a - Generality Multiple-choice. 
RIE    =   Remember - an - Instance Essay. 
RGE  =   Remember - a - Generality Essay. 
UGE  =   Use - a - Generality Essay.

If the general ‘F’ shows significance at (.05) priori level of significance, then a Post - hoc Analysis of Variance will be used by using a Scheffe test at ( .05 ) level of significance.

Instructional Task TOP
= = = = = = = = = = = =
The learning passage employed in this study was selected from the regular introductory course textbook "University English I". A - 913 - word passage, entitled "The Common Cold" contained 13 paragraphs discussed the idea of the common cold’s spread and ways of its prevention. Students who participated in this study haven’t had a previous knowledge of the passage. The passage mostly consisted of general information such as concepts and procedures, and specific information such as dates, names, and examples of the general information. The entire passage is presented in Appendix A (See, Appendix A).

The Experimental and Control Groups TOP
= = = = = = = = = = = = = = = = = = = = = = =
The sample of this study was divided randomly into six experimental groups and one control group. Each group consisted of (20) students. Three of the experimental groups were directed to read the passage once then answer a group of multiple-choice questions written on three different levels of learning based on Merrill’s Taxonomy ( RI, RG, UG). One group was directed to read the passage once, then to answer (7) multiple-choice questions of Remember - an - Instance level ( RI ) which required students to recognize specific information such as: Colds are caused by a: a) Germ, b) Bacterium, c) ýFangus, d) Virus. The second group was directed to read the passage once, then to answer (6) multiple-choice questions of Remember -a - Generality level (RG) which required students to remember general information such as: A contagious disease is a disease which a)doesn’t have treatment, b) spreads by touch, c) affects only pregnant women, d) causes death. The third experimental group was directed to read the passage once, then to answer (6) multiple-choice questions of Use - a - Generality level (UG) which required students to apply a generality to a new instance such as: Medicine can be given to patients in different compositions such as injection. Another common way of taking medicine is a) drinking water, b) tasting sugar, c) tablets, d) smelling air.

The other three experimental groups were directed to read the same passage and answer the same levels of questions which the above mentioned experimental groups have read and answered, but the questions were given to them in an essay type. The only difference between two of them was that the first three experimental groups received questions written in a multiple- choice form, whereas the second three experimental groups received the same questions but they were written in an essay form, that is, one group read the passage and answered (7) essay questions on Remember - an - Instance (RI) level, such as: What causes cold?). Another group read the passage and answered (6) essay questions on Remember - a - Generality level ( RG), such as: What is a contagious disease?). The third experimental group read the passage and answered (6) essay questions written on Use - a - Generality level (UG), such as: Medicine can be given to patients in different compositions like injection. Name another way of taking medicine that is commonly used by people.

The control group was directed to read the same passage once without receiving any type of questions. Each booklet was given a number to keep the track of the experimental and control groups when they took the posttest.

Measures TOP
= = = = = = = 
Based on Merrill’s taxonomy of learning (1983) a - 19 - item posttest was written on three levels of learning (Remember- an - Instance RI, Remember -a - Generality RG, and Use - a - Generality UG), asking about the general and specific information which mentioned in the passage (See, Darwazeh (1995a) to learn more about Merrill’s Theory). 10 of the 19 items were written in an essay form and 9 items, the rest of the posttest, were written in an objective form (See, Table 4).

The remember - level test items (RI and RG) were those which students have manipulated during the experiment, whereas the application level test items (UG) were different from those questions which students have manipulated during the experiment, otherwise they will be measuring the remembrance of application rather than the application level itself. The test tried to measure all the specific and general information that came in the passage. 

Table (4) shows the number of question types and levels which were used in the achievement test. 

Question Levels 
Question Types
RI 
RG 
UG 
Total 
Multiple-choice 
Essay 
10 
Total 
19 
Test Reliability TOP
= = = = = = = = = =

The reliability of the test was computed by using "Kuder Rirhardson 21". 

             x - { k - x }                    where : x = mean 
R = 1 - _________ 
             k (S . D .)                       S .D.= standard deviation

                                                   k = number of items on post- test 
                                                   (See, Shohamy,1985 p.21)

In order to use this formula, we computed the following: 

X= students’ scores = 1822.5 

X¯  = 1822.5 ÷  140 = 1

(x - x¯ ) ² = 4716.5

4716.5 ÷  140 = 33.7

S . D. = -/ ¯33.7 = 5.8

Thus, the reliability of the test is computed by the following steps 

       13 - ( 19 - 13 )
1 - _____________ =  ? 
       19 ( 5.8)

         78 
= 1 - ____ = .12 =  ? 
       639.16

Rx x  = 1 - .12 = .88

Procedure TOP
= = = = = = =

This study was conducted in the second semester of the academic year 1996/97, at An-Najah National University, Nablus, Palestine. The following procedure was followed:
1. The researcher selected the instructional passage, which is entitled "The Common Cold" from the required textbook University English I , at An- Najah University, after he coordinated with the teachers who teach the course . Students haven’t had a previous knowledge of the passage; they were about to take it, according to their lesson plan, within the coming ten days from the date of the experiment.

2. The researcher took the random sample of his study which consisted of (140) students and assigned them randomly into 7 groups: 6 experimental groups and one control group, each consists of 20 students.

3. For the six experimental groups, the researcher designed six booklets each consists of six to seven questions written in either an essay type or a multiple-choice type covering all specific and general information that came in the passage, and measuring three levels of learning based on Merrill’s taxonomy: Remember - an - Instance, Remember - a - Generality, and Use a Generality. For the control group, the same passage was only used with no post-passage questions (See, Table 5).

Table ( 5 ) shows the seven booklets which were designed for each of the experimental and control groups.
No. of the experimental groups whichwere used in the study Type & level of questions used   No. of questions used in each group
Experimental group no.1
RIM 
Experimental group no.2
RGM 
Experimental group no.3
UGM 
Experimental group no.4
RIE 
 7 
Experimental group no.5
RGE 
Experimental group no.6
UGE 
 6 
Experimental group no.7 (control group)
------ 
------ 

The seven booklets were shuffled together randomly before the experiment started in order to assure the random assignment of the sample of the study.

4. The researcher had also composed an achievement test to measure the students’ learning. The test consisted of 19 items that covered all specific and general information and that were written on three levels of learning based on Merrill’s taxonomy (RI, RG, and UG). These items were the same as the post-passage questions, except the Use - a - Generality questions which were different in order to avoid measuring the remember of application rather than the application level itself. 

Table (6) shows the number of question types and levels which were used in the achievement test.

Question Levels
Question types
RI 
RG 
UG 
Total 
Multiple-choice
Essay
10 
Total
19 

5. For the suitable time needed for the experiment, the researcher consulted the English teachers who teach the required course "University English I" to estimate the length of time that students at freshmen levels need to read the passage and answer the post questions, then the posttest. The researcher also conducted a pilot study to ensure the amount of time required by freshmen students. The time of the experiment was limited as shown in Table (7).

Table ( 7 ) shows the time distribution during the experiment .
Procedure 
Time allowed in minutes
Giving directions 
Papers distributions 
Reading the passage only 
25 
answering post-questions 
10 
collecting the booklets 
test distribution 
answering the test 
30
test collection 
Total Time
90 minutes 

The experiment took place during four consecutive session; 90 minutes for each, starting from 8:00 a.m. until 2:00 p.m., in one day. 
The researcher, who himself conducted the study, mentioned to the examinees the purpose of his study which was to investigate the effect of question types and levels on students’ learning in order to enhance students’ learning. The researcher also instructed the students to read and follow the instructions which were written on the front page of each booklet. Then, the researcher distributed the booklets within 5 minutes. Students were given 25 minutes to read the passage for once, and 10 minutes to answer the post- passage questions. The control group students were asked to read the passage once within 25 minutes and to wait for the test paper for 10 minutes doing something else like reading in other textbooks.

After the first 25 minutes, the researcher collected all the booklets within 5 minutes. Then, the researcher distributed the test within 5 minutes and reminded the students to read the general instructions which were written on the front page of the test. Students were instructed several times to write the numbers of the booklets on the top left side of their tests’ papers which they have just manipulated. Students were given 30 minutes to answer the posttest. After students finished answering the test, the researcher collected the papers test from all students at once. The same procedures were followed in the next three classes. Most students finished the experiment within 90 minutes.

Scoring TOP
= = = = = =
The researcher himself graded the post test blindly as students were not asked to write their names. A sample of the test forms was also re-scored by another researcher in order to establish the reliability of the scoring procedures. The scores of the test were as shown in the following table (8). The scores of each question item were distributed with the help of the study supervisor (See, Table 8). 

Table ( 8) shows the scores which were given to each sub-test questions.

Sub-Test Questions
Scores
Remember an Instance Essay Sub-Test (RIE) 
Remember an Instance Multiple-choice Sub-Test (RIM) 
Remember a Generality Essay Sub-Test (RGE) 
Remember a Generality Multiple-choice Sub-Test (RGM) 
Use a Generality Essay Sub-Test (UGE) 
Use a Generality Multiple-choice Sub-Test (UGM) 
TOTAL 22
22
TOP
1