shadow

SCIENTIFIC JUSTIFICATION OF SENTENCE LEVEL READING AND LISTENING QUESTIONS WITH 3 CHOICES USED IN THE TELS® EXAM


The reading and listening questions in the TELS® exam have been prepared at sentence level and with 3 choices to test the candidate’s ability to decode short written and oral content.

While testing these skills in equivalent exams (Toefl iBT, IELTS, PTE), long texts and speech content are used. This type of long content measures the candidate’s reading and listening strategies developed throughout their education process in their mother tongue rather than testing the pure ability of the candidate to decode the content provided in English. However, the purpose of the TELS® exam is not to test these strategies that already exist in the mother tongue, but to test the candidate’s ability to interpret and decode written and audio content prepared in English language, and to measure the knowledge and skill of pure language. For this reason, long texts are not included in the TELS® exam. Since its target audience consists of candidates with at least undergraduate degrees and it is assumed that these candidates have already successfully passed the reading exam in their native language in the university entrance exam, giving these candidates long texts, testing their reading strategies is unnecessary and can even create obstacles on testing their foreign language knowledge in a pure way since long texts increase the duration of the exam and the attention and perception of the candidate may decrease within time. In addition, these long texts may belong to certain fields of interest, and if the candidate is not interested in this field, incomplete or even high margin results about foreign language knowledge and skills can be reached.

Another important point is that in the above-mentioned equivalent exams, the change of question choices between 4 and 5 causes the candidate to have to read the non-functional choices in each question, and this may decrease his perception and attention over time, and decrease his performance for the remainder of the exam, thus causing misunderstanding about the actual language level of the candidate. As Haladyna and Downing (1989b) states, what matters is not the number of distractors, but the quality of the distractors (p. 59). Haladyna, Downing, and Rodriguez (2002, 318) argued that three choices were sufficient and added that four or five choices were unnecessary.

In addition, in 2008, Vyas and Supe conducted a research on the number of choices in the multiple-choice exams, and the results showed that there was no significant change in psychometric tests in 3-choice tests compared to 4 or 5, and even 3-choice increased efficiency. The reason for this was explained as fewer dispensers need to be prepared for tests with 3 choices, less space and less reading time. As a result, Vyas and Supe (2008, 130) concluded that 3-choice multiple-choice exams had the same quality as 4- or 5-choice exams, and therefore recommended 3-choice multiple-choice exams.

Again in 2011, Baghaei and Amrahi conducted another study on the number of questions in multiple-choice exams, and the result of this study was in a way that supports other studies. Given the amount of time and energy required for multiple choice tests with more dispensers, it was concluded that three choices per item are optimal (2011, 192).

Another study on the optimal number of questions was done in 2016 by Vegada, Shukla, Khilnani, Charan and Desai, and they came to the same conclusion. They recommended that there is no significant difference between the tests in terms of validity, reliability and item discrimination, so evaluation based on three-choice tests should be preferred over four or five-choice tests due to the advantage of three-choice tests.

Most of the studies on the number of questions conducted so far have found 3 choices optimal. According to Rodriguez (2005, 11), multiple choice exams should consist of three choices, one correct choice, and two reasonable possibilities. Using more choices does little to improve the item and test score statistics, and typically results in unreasonable odds. The Rodriguez study also showed that the threat of guessing and the risk of finding the right choice were not greater in 3-choice tests than in 4 or 5-choice tests (2005, 11). Kolstad, Briggs, and Kolstad (1985, 431) recommended that no more choices be used than are necessary to suppress prediction effectively.

That is why, in this article, a study has been conducted to support and strengthen the need for questions to contain a maximum of 3 choices, as supported by many scientific studies, and also that sentence-level questions can give healthy results about the candidate’s language skills, in order to test the pure ability of decoding reading and listening in English. The results came out to support the previous studies.

This study was conducted as a case study. The data collected was calculated in percentages and oral feedback was obtained with the groups participating in the study.

This study was carried out on 32 undergraduate students. The students have just passed the university entrance exam and are placed in a public university English Lannguage Preparatory Program for 1 year, consisting of 18 female and 14 male students aged between 18 and 20. 16 of these students are studying in one class and the other 16 are in another class.

In order for both groups to give homogeneous results, the questions in the study were prepared in Turkish, the native language of the students.

The first 16-person study group, female and male mixed, divided into 2 groups of 8 people, and the first group was given the following 5 choice question with several words missing. They were asked to read carefully and mark the correct answer:

The question given to the first 8 people in the test group:

Antarktika, ***** yüksekliği 2500 metre olduğu için ***** yüksek ***** ***** almıştır. Dağların  ****, sanıldığı gibi ***** ***** değildir. Dünyadaki ***** % 90’ı, tatlı suyun % 70’i buradadır. Burada ***** ***** 4000 metreyi geçer. Yıllık kar yağışı çok ***** ama karlar ***** ***** için biz onu hep karla ***** *****. Kışın ***** ***** -65 derece dolayındadır. Bu ***** ***** penguenler, *****, ***** ve ***** burada ***** *****.

Bu parçaya göre Antarktika’yla ilgili olarak aşağıdakilerden hangisi söylenebilir?

  1. A) ***** ***** karşın ***** bulunduğu
  2. B) ***** ***** olmasının yaşamı ***** etkilediği
  3. C) ***** arasında ***** ***** olduğu
  4. D) ***** ***** merak edildiği
  5. E) ***** ***** bulmada zorlandığı

The second group was given the following question and likewise asked to mark the correct answer:

Question given to the second 8 people in the test group:

Antarktika’da bulunan ***** ***** *****, penguenler, *****, *****, ve *****, burada *****.

Bu parçaya göre Antarktika’yla ilgili olarak aşağıdakilerden hangisi söylenebilir?

  1. A) ***** ***** karşın ***** bulunduğu
  2. B) ***** ***** olmasının yaşamı ***** etkilediği
  3. C) ***** arasında ***** ***** olduğu
  4. D) Doğal yapısının ***** *****
  5. E) ***** ***** bulmada zorlandığı

After both groups answered the questions, the papers were collected, and both groups were given the same question with the missing words placed back, and they were asked to mark the correct answer:

Question given to everyone in the test group, 16 people:

Antarktika, ortalama yüksekliği 2500 metre olduğu için en yüksek kıta unvanını almıştır. Dağların tepeleri, sanıldığı gibi buzla kaplı değildir. Dünyadaki buzun % 90’ı, tatlı suyun % 70’i buradadır. Burada buzun kalınlığı 4000 metreyi geçer. Yıllık kar yağışı çok değildir ama karlar hiç erimediği için biz onu hep karla kaplı görürüz. Kışın ısı ortalaması -65 derece dolayındadır. Bu koşullara rağmen penguenler, balinalar, foklar ve yosunlar burada yaşamlarını sürdürür.

Bu parçaya göre Antarktika’yla ilgili olarak aşağıdakilerden hangisi söylenebilir?

  1. A) Ağır koşullara karşın canlıların bulunduğu
  2. B) Buzla kaplı olmasının yaşamı olumsuz etkilediği
  3. C) Mevsimler arasında sıcaklık farkı olduğu
  4. D) Doğal yapısının merak edildiği
  5. E) Canlıların besin bulmada zorlandığı

(2010 YÖS Türkçe, sayfa 41 https://www.osym.gov.tr/Eklenti/2844,09052010yospdf.pdf?0)

As seen in the first two questions, some words in the question text, question root, and choices were removed, and how the candidates would answer in the absence of these words were tested. The aim was to show that candidates who have a lack of vocabulary and reading comprehension skills in foreign languages ​​have little chance to go to the correct answer even in the questions asked at the sentence level. The number of correct answers to the first question was 0, and the number correct answers to the second question was 2. The students who gave the correct answer to the second question were asked orally whether they marked it knowingly, guessingly, or randomly, and they stated that they marked it randomly. After both groups answered the questions given to them, the 3rd question, in which the missing words were placed back, was given to both groups at the level of the paragraph and all candidates were expected to answer this question correctly. The result was as expected and all of the students got the correct answer. The conclusion from this is that when the performance of the candidate is tested at the level of paragraph or sentence, there will be no difference between them, depending on the level of knowledge of the foreign language, and therefore, the reading and listening exams can be done at the sentence level and the results will give consistent results about the candidate’s knowledge and skills in the foreign language.

In order to achieve the second goal, 16 students in the control group were divided into two groups of 8, in the same order,  and given the same questions with weak distractors eluded, which were deemed unnecessary. The questions are given as follows, respectively:

Question given to the first 8 people in the control group:

Antarktika, ***** yüksekliği 2500 metre olduğu için ***** yüksek ***** ***** almıştır. Dağların  ****, sanıldığı gibi ***** ***** değildir. Dünyadaki ***** % 90’ı, tatlı suyun % 70’i buradadır. Burada ***** ***** 4000 metreyi geçer. Yıllık kar yağışı çok ***** ama karlar ***** ***** için biz onu hep karla ***** *****. Kışın ***** ***** -65 derece dolayındadır. Bu ***** ***** penguenler, *****, ***** ve ***** burada ***** *****.

Bu parçaya göre Antarktika’yla ilgili olarak aşağıdakilerden hangisi söylenebilir?

  1. A) ***** ***** karşın ***** bulunduğu
  2. B) ***** ***** olmasının yaşamı ***** etkilediği
  3. C) ***** ***** bulmada zorlandığı

Question given to the second 8 people in the control group:

Antarktika’da bulunan ***** ***** *****, penguenler, *****, *****, ve *****, burada *****.

Bu parçaya göre Antarktika’yla ilgili olarak aşağıdakilerden hangisi söylenebilir?

  1. A) ***** ***** karşın ***** bulunduğu
  2. B) ***** ***** olmasının yaşamı ***** etkilediği
  3. C) ***** ***** bulmada zorlandığı

After both groups answered the questions, the papers were collected, and both groups were given the same question with the missing words placed back, and they were asked to mark the correct answer:

Question given to each of the 16 people in the control group:

Antarktika, ortalama yüksekliği 2500 metre olduğu için en yüksek kıta unvanını almıştır. Dağların tepeleri, sanıldığı gibi buzla kaplı değildir. Dünyadaki buzun % 90’ı, tatlı suyun % 70’i buradadır. Burada buzun kalınlığı 4000 metreyi geçer. Yıllık kar yağışı çok değildir ama karlar hiç erimediği için biz onu hep karla kaplı görürüz. Kışın ısı ortalaması -65 derece dolayındadır. Bu koşullara rağmen penguenler, balinalar, foklar ve yosunlar burada yaşamlarını sürdürür.

Bu parçaya göre Antarktika’yla ilgili olarak aşağıdakilerden hangisi söylenebilir?

  1. A) Ağır koşullara karşın canlıların bulunduğu
  2. B) Buzla kaplı olmasının yaşamı olumsuz etkilediği
  3. C) Canlıların besin bulmada zorlandığı

The number of students who did the first question correctly was 0, and the number of students who did the second question correctly was 3. The students who gave the correct answer to the second question were asked orally whether they marked it knowingly, guessingly, or randomly, and stated that they marked it randomly. After both groups answered the questions given to them, the 3rd question, in which the missing words were completed, was given to both groups at the level of the paragraph and all candidates were expected to answer this question correctly. The result was as expected and all of the students got the correct answer. The conclusion from here is that when non-functional distractors are removed from the 4 or 5-choice questions and the number of choices is reduced to 3, the ratio of students to answer a question correctly does not change. For this reason, we can conclude that testing the foreign language knowledge and skills of the candidates with an exam consisting of 3-choice questions will give a similar result compared to exams consisting of 4 and 5-choice questions. In addition, the number of correct answers to the first 5-choice single sentence question and the second 3-choice single sentence question were 2 and 3, respectively. Here, we can conclude that the threat of guessing did not make a significant difference in 5 or 3-choice questions. Therefore, this result also supports the results of the aforementioned study.

As a result, it was seen that this study supports the judgment that 3-choice sentence-level questions can be used in the preparation of reading and listening questions to test foreign language knowledge and skills, and the TELS exam is shaped according to the reading and listening questions that are formed at sentence level with 3 choices.

 

REFERENCES

Asmus, E. J., Jr. (1981). The effect of altering the number of choices per item on test statistics: Is three better than five?. Bulletin of theCouncil forResearch in Music Education, 54, 948–950.

Baghaei, P., & Amrahi, N. (2011). The effects of the number of options on the psychometric characteristics of multiple choice items. Psychological Test and Assessment Modeling, 53(2), 192-211.

Haladyna, T. M., & Downing, S. M. (1989b). Validity of a taxonomy of multiple-choice item writing rules. Applied Measurement inEducation, 2, 51–78.

Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002). A review of multiple-choice item-writing guidelines for classroom assessment. Applied Measurement in Education, 15, 309–334.

Kolstad, R. K., Briggs, L. D., & Kolstad,R. A. (1985). Multiple-choice classroom achievement tests: Performance on items with five vs. three choices. College Student Journal, 19, 427–431.

Rodrigues, M. C. (2005). Three options are optimal for multiple-choice items: A meta-analysis of 80 years of research. Educational Measurement: Issues and Practice, 24(2), 3-13.

Vegada, B., Shukla, A., Khilnani, A., Charan, J., & Desai, C. (2016). Comparison between three option, four option and five option multiple choice question tests for quality parameters: A randomized study. The Indian Journal of Pharmacology, 48(5), 571–575.

Vyas, R. & Supe, A. (2008). Multiple choice questions: A literature review on the optimal number of options. Medical Education, 21(3), 130-133.