Clara Román-Odio & Bradley Hartlaub

The Ohio5 group, like many language educators in this country, believes that multimedia-based approaches offer significant advantages over traditional teaching methods by accelerating and sustaining the process of language acquisition. Our success in procuring long-term support that will nurture new advances in the field depends on the extent to which we can articulate in quantitative and objective terms the pedagogical benefits provided by these new technologies.

The primary objective of this work is to describe the major steps involved in designing a comparative study that provides a rigorous and systematic evaluation of the use of multimedia in the language classroom. We hope this initiative will help others in the Ohio5 to plan and carry out studies that will yield new and reliable information about the impact that these technologies are having in language learning.

The work will address basic questions regarding methods of assessment in language teaching, including the use of statistical design and analysis, such as:

• Where do I start?
• What instruments am I going to use to assess student learning?
• How can I ensure the validity and reliability of my assessment instruments?
• Why do I need randomization?
• How do I form a control group?
• Where should I turn for useful language assessment and statistical resources?

It will also discuss the importance of key aspects of experimental design including: sample size determination; eliminating bias; testing techniques to assess writing, oral, reading and listening skills, grammar and vocabulary; statistical significance; and examples of assessment instruments (quizzes, student evaluations, minute cards, etc.). The methods will be illustrated by a case study describing Román-Odio and Hartlaub’s assessment of Kenyon’s Spanish music program, Ingenio, and by a sample of successful studies intended to measure qualitative and quantitative effects of CMI (Computer Mediated Instruction) on language learning. The final section of the document will provide a summary of recommendations made by CALL (Computer Assisted Language Learning) researchers on assessment of educational technologies in the languages.

I. Where Do I Start?

As with any research situation, the very first step is to state as clearly as possible what you want to know and for what purpose. Without a clear statement of the research question it is difficult to collect useful information and make appropriate inferences. The following questions can help you draft a research statement:

• What is the precise purpose of this study?

• What language abilities do I want to test?
• What technologies and / or teaching methodologies do I want to assess?
• What constraints are set by unavailability of expertise, time, baseline data?

Once the research statement is clear, then steps can be taken to turn your research question into an evaluation plan. One way to begin expressing such plan in concrete terms is by outlining specifications for a hypothesis test. Arthur Hughes suggests a useful framework for test construction, including the following categories:

A. Framework for Test Construction:

Research Statement: For example, I created a nifty multimedia-based program for my second and third year Spanish students. The program has . . . . The students learn . . . . I wonder whether this is an effective way to help students develop their . . . (speaking / writing / reading / listening / oral) ability. To answer this question, I would like to study the effects of . . . . (technology / methodology) on the development of the (speaking / writing / reading / listening) ability of my students.

Operations: The tasks that students have to be able to carry out. For a reading test, for example, these might include: the ability to scan a text to locate specific information, construe the meaning of complex passages, guess the meaning of words from context, identify the referents of pronouns, etc. For a speaking test it could be giving directions.

Types of text: For a reading test these might include passages from textbooks and newspapers.

For a writing test they could be letters, post cards, notes, descriptions, narrations, short essays.

Addressees: These are the people to whom the student is expected to be able to write or speak. For example, native speakers of the same age and peer group.

Topics: The subject areas reflect the kinds of written and spoken texts that are to be found in the teaching materials at various levels.

Format and Timing: This aspect should specify test structure and / or type of treatment, including time allocated to components and elicitation procedures, with examples. It should include how many passages will be presented (for reading or listening) or required (for writing), the number of items in each component and what weighting is to be assigned to each component. It should also include a description of the treatment applied to the experimental and the control groups.

Criteria to Determine Levels of Performance: This refers to the required level (s) of performance for different levels of success. It may consist of a simple statement to the effect that to demonstrate mastery, 80 percent of the items must be answered correctly. It may also be more specific using, for example, descriptors from the ACTFL (American Council for the Teaching of Foreign Languages) scales or from other methods of scoring such as the analytical method devised by John Anderson.

Scoring Procedures: Scoring procedures should be specified. There could be a detailed key, making scoring almost entirely objective, or there could be multiple scorers where scoring is subjective. According to Hughes, scoring will be valid and reliable if:

• a description of criteria levels is written and scorers are trained to use them

• irrelevant features of performance are ignored (for example, spelling in a reading test)

• there is more than one scorer for each performance (86-97).

Sampling: For content validity, pre-testing and beneficial backwash (the effect of testing on teaching and learning) Hughes recommends the creation of several versions of the test. The best way to identify items that are faulty is through teamwork. Your colleagues can be an excellent resource to identify items in your assessment plan that need improvement. Critical questions that may be asked include:

• Is the task perfectly clear?

• Is there more than one possible answer?
• Can students arrive at the correct response without having the skill being tested?
• Do students have enough time to perform the task?

In addition, one version of the test should be administered to native speakers (who should score 100 percent or close to it) of a similar educational background of your students.

Pre-testing: It can help you identify problems in administration and scoring. It is also essential to be able to calculate the reliability coefficients of the test or treatment (Hughes 49-52).

Clearly, Hughes’ framework is theoretical, specifying only main facets of testing. In order to turn your research question into an evaluation plan it is essential to define the behavior or characteristics that could reasonably constitute the variables that you are interested in assessing. The Follow-up Activity sections of this work are intended as an opportunity to practice the process whereby an assessment question is operationalized. Take a few minutes to read them and jot down your ideas.

B. Follow-up Activity for Developing an Assessment Plan

As part of an assessment team effort you have been asked to write a draft of the research statement, operations and format-timing portions of the evaluation plan for the following situations:

1) A college Russian teacher has developed a series of digital movies, captioned with subtitles in the target language, for her first-semester students. The digital movies, which include authentic video clips, were created with QuickTime software. Students can view them with the captions or without. She wants to know what effect captioning has on listening-comprehension skills (Collentine & Sweeney, "An Introduction to Conducting CALL Research" CALICO 99 Workshop, June 2, 1999).

2) Our French colleagues asked their students to dialogue in the target language with other classmates and other French classes via chatrooms. We would like to know if this is an effective way to help students develop their speaking abilities. How can we measure this effect? What type of treatment (conditions / methodologies) can we apply to demonstrate the efficacy of this technology-based teaching practice?

II. What Instruments am I Going to Use to Assess Student Learning?

This will depend on two factors:

• What language skills do you want to test

• What type of technology you are using to enhance such skills

We make the assumption that the best way to test, for example, students’ writing abilities is to ask them to write. Since the same applies for oral, reading and listening abilities, we will discuss testing techniques for each skill separately. The most important principle to keep in mind is that we want to set tasks that form a representative sample of the tasks that we expect our students to be able to perform. To identify and select a representative sample of such tasks we can use Hughes’ framework presented above. For example:

A. Writing:

• purpose: to test adequacy of student’s written Spanish for participation in a study abroad program

• operations: describe, request, explain, compare and contrast, argue

• text type: up to two paragraphs in length

• addressees: native and non-native university lecturers

• topic: any capable of academic treatment

• format: complete a job application; write a post card to a friend requesting a favor and apologizing for any inconvenience; write to a Spanish bookstore to request information about a good dictionary (qualities, price, method of payment); discuss the benefits and drawbacks of a college education in English.

To obtain samples that properly represent the student’s writing ability Hughes offers the following recommendations.

1) Set as many tasks as possible so that you offer students as many fresh starts as possible. The more scores you have for each student the more reliable the total score should be (81).

2) Test only writing and nothing else. You don’t want to test if the student is creative, imaginative, intelligent or has a good reason to hold an opinion. One way of reducing this risk is to make use of visual materials. For example:

• a series of pictures can be used to elicit narration;

• three objects can be lined-up for comparing and contrasting;

• graphs can be used to ask students to explain and support a series of conclusions.

3) Restrict students’ arguments to keep them from going far astray. This will make comparisons between students’ writing abilities easier.

4) Determine a method to obtain reliable scoring for writing. Scoring can be holistic (based on an overall impression or on descriptors of exiting scales such as the ACTFL scale for writing) or analytical, requiring a separate score for each of a number of aspects of a task (81-97).

5) Follow-up Activity for Assessment of Writing Abilities

1. Design a writing exercise that can be performed both in the classroom and in the computer room. Following the advice given above, develop an assessment plan to determine the effect of CMI on your students’ writing skills. You may ask, for example:

• Are students more likely to take risks in a CMI environment than in the classroom?

• What writing ability markers would I include to respond to this question?

• In which of the two environments are students more reluctant to make use of dictionaries or other reference sources?
• Are there score differences between the two groups?

B. Oral Ability

Oral ability, or the ability to interact successfully in the target language, requires speech comprehension and production. To test oral ability, as for writing, we want to choose tasks that form a representative sample of the tasks that we expect our students to be able to perform. To specify such tasks we can make use of the ACTFL rating scales for oral proficiency. Thus, if we test at the intermediate level, our test operations may include the language functions presented in the ACTFL description of the Intermediate-Low / Mid / Advanced ability. Once the operations or tasks are established we can select a format and elicitation techniques. For example:

1) Format:

• interview, interaction with peers, or response to an audio or video tape

2) Elicitation Techniques:

• questions and requests for information

• pictures (for description and narration)

• role playing by two students

• dialogue exchange to make a decision

• situation description (e.g. "you are walking down the street and meet a friend who you thought was living in Spain. What do you say?")

3) Follow-up Activity for Assessment of Oral Abilities

a) Using passages (oral interactions) based on a captioned digital movie, construct a series of tasks to predict oral ability. You may use, for example, cloze passages, short answer questions, or narrations. Repeat the exercise using the same passages, but to be presented on a tape recording. How can you demonstrate whether the technical variables (captioned digital movie vs. audio

tape) have a different effect on the oral ability of your students? Write a draft of the research statement, operations and format-timing portions of an assessment plan to answer this question.

C. Reading Ability

1) In order to select a representative sample of the reading abilities that you want to test, it is necessary to specify them as accurately as possible. What are the macro-skills directly related to reading abilities?

• Scanning text to locate specific information

• Skimming text to obtain the gist

• Identifying stages of an argument

• Identifying examples presented to support the main arguments

2) Underlying these are other micro-skills including:

• Identifying referents of pronouns

• Using context to guess meaning of unfamiliar words

• Recognizing indicators in discourse, especially for introduction, development, transition and conclusion of ideas.

3) The types of texts you select (e.g. newspaper editorial, letter, poem, timetable, 2,000 words from a novel or an essay) will depend on the macro-skill you want to test. For example, to test scanning you will need passages which contain plenty of discrete pieces of information. Detailed reading, on the other hand, can be tested using a few sentences. Assuming that what you want to test is only reading ability, Hughes recommends to choose texts which students have not read, or texts that are not too culturally specific. Once the texts and tasks have been selected you can choose elicitation techniques, such as:

4) Elicitation Techniques:

• multiple choice (preferably to choose a picture describing a sentence)

• unique answer (a single word or something slightly longer)

• short answer (one sentence)

• guided short answer (framing the item so that students complete only what is presented)

• summary cloze (a reading passage is summarized with gaps to be completed)

• identifying order of events and arguments (using a column next to arguments and N/A)

• identifying referents (What does the word "it" (line 25) refer to?)

• guessing the meaning of words from context (e.g. find a word in the passage between lines 1-20 which means . . . )

Keep in mind that the elicitation techniques should make minimal demands on the writing abilities of your students, and that errors of grammar, spelling or punctuation should not be penalized (as in a listening test) since this will make the measurement of reading ability less accurate..

5) Follow-up Activity for Assessment of Reading Abilities

a) Can you think of one or two ways by which you could measure the effect that CMI may have on the development of your students’ reading abilities? Are there software mechanisms which can effectively promote reading abilities? Design a series of computer-based tasks to respond to this inquiry. You may consider, for example, the following alternatives:

• voiced + written text vs. silent reading for recalling main arguments

• highlighting unfamiliar words for retaining new vocabulary

• highlighting indicators in discourse for identifying order of events and arguments

• annotated vs. not annotated texts for overall comprehension (using summary cloze)

• marking all referents in the text

How would you incorporate these features into your research question?

D. Listening

Because listening (like reading) is a receptive skill, it parallels in many ways the testing of reading. As with reading, we should first specify the macro and micro skills directly related to listening abilities, including:

1) Macro-Skills:

• listening for specific information

• obtaining gist of what is being said

• following directions

• following instruction

2) Micro-Skills:

• interpretation of intonation patterns (for recognition of sarcasm or surprise, etc.)

• recognition of function structures (such as interrogative as request, e.g. Could you pass me the salt?)

3) The types of texts you select will depend, once again, on the macro-skill you want to test. For example, if you are interested in how students handle language intended for native speakers, then you should use samples from authentic speech (e.g. radio broadcast, T.V. news, dialogue from a soap opera or from a multi-participant talk show). If, on the other hand, you want to test the students’ abilities to follow an academic lecture, you may need a native speaker reading from a longer text. The elicitation techniques will also depend on the objective of the test. For example:

4) Elicitation Techniques:

• information transfer (to report a car accident / showing routes on a map, or labeling diagrams or pictures)

• multiple choice (choose a picture describing a situation)

• short answer (requiring minimal writing)

• note taking (students take notes during a talk and only at the end see the items to which they have to respond)

• partial dictation / transcription

5) Follow-up Activity for Assessment of Listening Abilities

a) The Spanish faculty members at your institution are using a new video program in their beginning Spanish classes which includes traditional and digital video (Quick Time). They would like to determine whether digital video is more efficient than traditional video technology in developing students’ listening skills. Following the advice offered above, design a series of tasks to respond to this question.

E. Grammar and Vocabulary

While grammar and vocabulary are essential to the development and demonstration of communicative skills, they are rarely regarded as ends in themselves (Hughes 150). Most of our final exams, however, test these underlying abilities in some way. Ideally, in testing grammar, students should supply the appropriate grammatical structures through the use of paraphrase, completion, and modified cloze. Regarding vocabulary, if it has been consciously taught, then all the words presented should be included in the specifications. Otherwise, words can be grouped according to their frequency and usefulness. Definitions, synonyms, gap filling, and matching words with pictures constitute the most frequently used testing techniques for vocabulary. In this area it would be useful to investigate whether computer-based instruction of grammar (encouraging form-focus) and of vocabulary are more effective than the teacher-based instruction.

III. Experimental Design

The most useful resource we have found for providing a starting point for any researcher designing a new experiment is A Checklist for Planning Experiments in Dean and Voss (1999) (Appendix A).

A. Blocking, Randomization and Replication

While we are certainly not intending to write a complete introduction to design and analysis of experiments in this document, we would like to briefly mention three primary techniques that are fundamental to experimental design: blocking, randomization and replication.

1) Blocking is when you purposely design your study to reduce the variability due to a known source of variation (called a block). For example, many agricultural studies use fields as blocks because it is well known that soil type and fertility vary from location to location. To try to reduce this source of variation the treatments (perhaps fertilizers) are randomly applied within each field, which has similar soil type and fertility.

2) When we say randomly in design we do not mean haphazard or with no apparent structure, we mean the deliberate use of chance in a planned way. While this step may seem unnecessary and time consuming, it is the most important and most overlooked step in the entire process. The primary reason for using randomization is to reduce bias. Although there are many types of bias, two of the most common are researcher bias and systematic bias. Researcher bias refers to the bias that results from a researcher using their personal opinions to influence, either deliberately or unintentionally, any aspect of the experiment. Systematic bias refers to a bias that often results from the order in which the treatments are applied. For example, a researcher may decide to conduct all trials for treatment 1 and then all trials for treatment 2. However, if the measurements are collected by laboratory technicians, the technicians may improve as they continue to apply the experimental procedure or they may tire from repeated application of the same task. Either way, any comparisons of the responses from treatment 1 with those from treatment 2 are confounded (mixed up) with the order in which the observations were collected. (Refer to Appendix B for an example).

3) The primary importance of replication is to insure precision. Replication refers to fact that many experiments are repeated under the same conditions on different experimental units so we can estimate the variation within a set of experimental conditions and also compare the variability across differing experimental conditions. Both sources of variation (variation due to error and variation due to the treatments) need to be estimated for a complete analysis. It is important to distinguish replication from repeated measures. Replication refers to settings where independent pieces of information will be collected each time the experiment is conducted. Repeated measures refers to settings where multiple observations are recorded for the same experimental unit (e.g., recording student performance in several testing periods).

B. Control and Experimental Groups

In all comparative experiments it is essential to have a comparison group, typically referred to as a control group. The control group is formed and treated the same as the experimental group, except these experimental units do not receive the treatment of interest. They may receive a placebo or some form of a standard treatment we are trying to compare with, but they do not receive the treatment of interest in the experiment. The idea is to form two identical groups that differ in only one way, the treatment of interest.

C. Matched Pair Design / Paired Replicates Design

While trying to form two groups of identical students sounds like a reasonable idea, it is not as easy as you might think. Therefore, rather than using a completely randomized design, it is sometimes preferable to use a matched pairs or paired replicates design. In a matched pairs design, pairs of subjects from the population are formed so that members of a given pair are as similar as possible with respect to one or more characteristics that might potentially affect the measurement to be investigated. Identical twins are commonly used for matched pairs designs on human subjects. In language studies we can form pairs of students by ranking them according to their language abilities. One member of each pair will be randomly assigned to the treatment group and the other to control group. For a paired replicates design, subjects serve as their own controls in providing baseline measurements prior to being exposed to the experimental set of circumstances or treatment and then they are measured once again after such exposure. Differences between the pre and post (or before and after) observations are then used to assess the effect of the treatment.

D. Sample Size Determination

Two different methods are commonly used to determine the appropriate sample size or number of observations needed on each treatment. Both methods require some specific information that will often require a pilot study. For example, we must have an estimate for the amount of variability due to experimental error. How much variation do we expect to see in the exam scores? We must also decide how confident the researcher wants to be in their results. Since it is impossible to be 100% certain of the outcome in a random experiment, we must be willing to accept some chance of error. Most journals require 95% confidence, or a 5% error rate, to publish manuscripts. However, it is also reasonable to require 96, 98, or 99% confidence in certain situations. The third piece of information required for sample size determination problems is some idea of what practical differences are for the given situation. For example, if we are giving a proficiency test and scores in the 80's demonstrate a certain level of proficiency while scores in the 90's demonstrate an improved level of proficiency then we may want to design our study to detect differences of size 10. For more algorithmic approaches to calculating sample sizes, see Dean and Voss (1999).

There are several important points to keep in mind when determining the sample size.

1. The more confident you want to be, the more observations you will need.
2. The algorithms used by statisticians for determining sample sizes will not include budget or time restrictions of the experiment.
3. To reduce the sample sizes you may have to reduce the confidence level slightly, refine the experiment in order to reduce the amount of error variability, or allow for less precision (wider interval estimates).

IV. Statistical Data Analysis

A. Descriptive Statistics and Graphs

A thorough statistical analysis of your experiment should include descriptive statistics, graphical summaries, and formal decision rules (known as hypothesis tests) to test your hypotheses. Common descriptive statistics that are used to measure the center of a distribution are the mean and the median. The mean is the average of the observations and the median is the "middle" observation. We placed middle in quotes because the actual computation of the median depends on whether you have an even or odd number of observations. If you have some unusual observations in your data, then the median is the preferred measure of center.

Two common visual displays that can be used to provide a graphical summary of your data are histograms and a boxplots. Histograms are formed by partitioning the range of the data into small intervals and finding out how many of the observations fall into each of these intervals. Bars are then erected over the intervals. The heights of the bars are frequencies, relative frequencies, or percentages. Boxplots are constructed from five descriptive statistics (minimum, 25^th percentile, median, 75^th percentile, maximum) and they are extremely useful for comparing two or more sets of observations. For example, comparing scores of the control group with scores of the treatment group.

Although these exploratory data analysis techniques are extremely useful and should be included in any analysis, many journals require formal statistical inference for publication. The most commonly used vehicle for completing the inference is a hypothesis test. Moore and McCabe (1999) provides a complete introduction to hypothesis testing, but the major elements of a hypothesis test are summarized below.

B. Hypothesis Testing Definitions

C. Hypothesis Testing Procedure

This will involve ensuring that your assessment instruments are reliable and valid, using multiple scorers and providing a detailed scoring key at the outset of scoring.

1) Test Reliability

If a test measures consistently it is considered reliable. There are standard procedures to estimate a test’s reliability (Henning 80-87). The first requirement is to have two sets of scores for comparison. The most common method of obtaining such scores is the split half method. This method involves one administration of one test that is split in two halves which are really equivalent, through a careful matching of items (Henning 83). Each student is given two scores. If the two scores are strongly correlated the test may be considered reliable. It is possible to quantify the reliability of a test through a mathematical measure of similarity known as reliability coefficient. Perfect agreement between two sets of scores will result in a reliability coefficient of one. Total lack of agreement will result in a zero. Thus, test reliability coefficients will range from 0 to 1. (Refer to Appendix E for procedure).

2) Test Validity

A test is said to be valid if it measures accurately what it is intended to measure (Hughes 22). How do we know that a test is really measuring what it is supposed to measure? Test specifications and representative samples of the functions that are meant to be covered (as shown in part II of this work) are essential to ensure content validity. Pre-testing can also help you determine the validity coefficient of a test (for a better understanding of the value of this procedure see Hughes 23-24). Finally, keep in mind that the best test can give invalid and unreliable results if is not carefully administered. For a more in depth understanding of the concept of validity see Chapelle’s Internal and External Validity Issues.

3) Scorer Reliability and Scoring Keys

Another aspect of eliminating bias is related to scoring. If no judgement is required on the part of the scorer, the scoring is objective. If judgement is required, multiple scorers are recommended. With the holistic method it is essential to identify benchmarks that typify key levels of ability. Only when there is an agreement on these benchmarks should scoring begin. Scorers should also specify acceptable answers and assign points for partially correct responses. (For a detailed explanation of holistic, analytical and objective scoring methods see Bailey 185-203).

V. A Case Study of a Multimedia Language Program

A. Objective

The purpose of this study was to investigate the effects of CD dictation and multimedia-based text on the listening and comprehension skills of 25 students enrolled in an introduction to Latin American literature and popular culture course. The study also inquired about the students’ response to using CD dictation and Ingenio Romance y Protesta en la Música Popular Hispana, a music-based multimedia program developed by the Spanish faculty at Kenyon College.

B. Research Background

Previous research on music and language acquisition indicates that music and language are closely related, both neurologically and developmentally (Crooswhite, 1996; Homburg, 1980; Levman, 1992; Zierer, 1985). Some researchers theorize that music and language follow similar developmental patterns in adjacent areas of the brain (Radocy & Boyle 174). But why use music in the foreign language classroom? Based on a study of students enrolled in a beginning French course at the University of Florida, Leutenegger and Mueller (1964) suggest that musical aptitudes--pitch, loudness, rhythm, time, timbre, and tonal memory--might be important in foreign language learning, with tonal memory the most significant predictor. Abrate (1983) lists other benefits that accrue through music, including the fact that songs:

• hold the attention and interest of students;

• introduce native and colloquial language, as well as pronunciation variants;

• present cultural phenomena and points of view;

• provide a context for language learning;

• offer an entertaining alternative to textbook study.

Wilcox (1985) further proposes that music may aid language learners in acquiring the prosody (rhythm, tone and stress patterns) of the target language. This seems to be mediated by:

• the creation of a prosodic framework, which increases recall and retention of vocabulary and may affect pronunciation in a positive way (10);

• the melody, which may aid the learner in direction and focus for retrieval from memory (6)

• residual learning, or the involuntary rehearsal of a song long after the music stops (46).

Music also has a therapeutic value. As Graham (1989) explains, music "can be used to establish a mood, to lessen anxiety, to encourage calmness, to ease loneliness and soothe irritability" (3). Furthermore, the language of emotions is commonly encoded in songs and this language can support the acquisition of the target culture. Once again, to cite Graham: "the vocabulary fluency people learn in song allows a fluency of emotion and a communication of their most sincere feelings" (3).

In view of this background, this study seeks to explore the potential benefits of using music via multimedia, by examining the relationship between aural / visual exploration of key cultural values and language learning. CALL researchers have indicated that the simultaneous presentation of language through multimedia can substantially improve student language achievement. Focusing on aural input, for example, Mann (1995) reports that hypermedia learning environments are particularly effective when text is accompanied by sound. Leow (1995) confirms this claim by showing that language learners are more likely to notice language structures when they are presented as aural input, or when text and aural stimuli are presented simultaneously. Research on the effect of captioning (where video, sound and text concur) supports a similar claim by demonstrating that the textual modeling of the captions have a significant impact on the comprehension and oral abilities of the students (Garza, 1991; Borras, 1995). In the area of colorization, investigators have shown that highlighting a target structure can enhance not only listening and reading skills but productive skills as well (Bell, 1984; Garza, 1996). The research question underlying this study is whether the simultaneous presentation of spoken (songs) and written (texts) language through multimedia has an effect in the listening and comprehension abilities of the student.

C. Method

Using each student as his / her own control, the study traces the development of the listening and comprehension skills of 25 Spanish students during twelve 1 ½ hour training periods. Songs were randomly assigned to 12 time slots. During periods 1-4 students received traditional instruction, which consisted of listening to and analyzing songs. During periods 5-12 CD dictation and the cultural contexts of the two musical genres being studied--salsa and nueva canción--were introduced using multimedia. Salsa was taught in periods 1, 2, 5, and 8 and nueva canción in periods 3, 4, 9 and 12. At the end of each period, students’ listening and comprehension skills were tested using a cloze exercise of song transcription and three short answer questions. All of the tests were administered in the same way each time and had the same format: 30 fill-in-the-blank items based on a song (10 nouns, 10 verbs, 10 adjectives or adverbs) and three short answer questions. CD dictation allows students to transcribe a song from a CD and to see the song text through the feedback feature of the program. Ingenio combines text, music, images, video and interactive exercises in the teaching of several genres of Hispanic popular music. The effect of the new instructional method was evaluated in periods 6, 8, 10 and 12, using the same type of cloze and short answer exercise employed during the first four periods of training. There were eight testing session (periods 1-4 and periods 6, 8, 10 and 12) and students did not know that the study was being conducted. By the end of the experimental period we had approximately 440 scores (16 scores for each of the 25 students). Formal hypothesis tests were used to analyze the data and a questionnaire was administered to gauge students’ response to using CD dictation and Ingenio in the course.

D. Results and Discussion

1) Listening

A formal statistical analysis of the data indicates that there is a significant increasing trend (t=8.23, p=.0001) in the listening scores for the first four sessions where the traditional method was used, as well as for the last four sessions (t=8.44, p=.0001) where the new method was used. In the last four sessions, however, the average listening scores are significantly higher than the scores in the first four sessions (5.620 vs. 6.863). This suggests that the listening skills of students improved significantly with the introduction of multimedia and CD dictation. Nevertheless, a caveat that must be kept in mind when interpreting these results is the difficulty of discriminating time-dependent from multimedia-associated improvements. In a future experiment this limitation may be overcome by a randomized alternation of multimedia and traditional methods of instruction, or by randomly assigning students into two groups--with multimedia and without multimedia.

In order to interpret our results it is essential to ask what is the precise object of our testing. In assessing students’ listening skills we are not testing what vocabulary they acquired, nor if their pronunciation or their ability to distinguish between the letters c / z / s improved after listening to songs. Indeed, the only question that our assessment instrument or test can answer is whether CD dictation and multimedia have a significant effect on attentive listening and speed writing. In our testing, each song represents a fresh start for the students. Therefore, over time students learn to pay a higher level of attention to the song language. Only through a fine tuning of this micro-skill they become able to transcribe fast enough to keep up with the song. Anyone who has attempted to transcribe a song written in a foreign language knows the time and the difficulty involved in this process. Our question is, then, can students truly develop attentive listening through multimedia? CD dictation, the training tool that we used, allows students to listen to, type and see the song language. The results of our study (the listening scores in the last four sessions are substantially higher than those of the first four sessions) suggest that by simultaneously engaging the aural and visual sensorial receptors of students, this training tool boosted the students’ ability to understand and transcribe songs. Previous research supports the claim that CALL applications can facilitate and advance various language skills simultaneously by engaging the speaker in integrative tasks (i.e. requiring listening, reading, speaking and writing abilities). In view of these findings, maybe the most significant conclusion of the listening aspect of our study is that multimedia may help teachers to target areas of language skills in such a specific and concurrent way (i.e. attentive listening and speed writing) that students’ overall foreign language development is inevitably achieved.

2) Comprehension

Comparing the comprehension scores for the first four sessions with those in the last four sessions, we find that the average scores are significantly higher (8.385 vs. 8.944) in the last four sessions. There is also a significant increasing trend (t=4.55, p=.0001) in the scores during the first four sessions, while there is not a significant linear trend (t=1.64, p=.1032) in the scores during the last four sessions. Thus, the results indicate that, although overall students’ comprehension skills improved significantly, they reached a plateau during the last three testing periods of the experiment. In order to explain this outcome we need to take a closer look at the overall trend in the scores, for which we used a scatterplot smoothing method. The overall increasing trend is clearly visible for the listening scores in Figure 1. For the comprehension scores in Figure 2 we observe a significant increasing trend, culminating at test 5, and a subsequent plateau.

This can be explained as the result of at least two confounding factors related to the assessment instrument used to test comprehension. First, it is possible that in answering the same type of content-based question (analysis of theme, rhetorical elements and musical components of the song), the students learned how to respond in order to obtain a higher score. Second, the comprehension questions referred exclusively to the song being tested and, therefore, do not reflect students’ acquisition of overall comprehension and cultural competence obtained after introducing Ingenio. This aspect was tested in the course’s final exam but was not part of this experimental design and data analysis.

Boxplots were used to compare the distributions of the scores for the eight testing session. Figure 3 contains side-by-side boxplots of the listening test scores.

Notice that the overall trend is increasing, but students did not do as well on tests 3 and 5. A careful examination of these two tests, as well as comments made by students, suggest that song #3 was particularly challenging due to drastic changes in pitch voice, rhythm and musical arrangement. The low scores of test # 5 seem to be the result of a difficulty in dealing with new vocabulary. Figure 4 contains side-by-side boxplots of the comprehension scores.

The comprehension scores tend to be higher than the listening and therefore pattern over time is not as clear. The use of other elicitation and testing techniques to assess comprehension (such as recording an oral statement from the students based on the song, identifying the order of events presented in the multimedia program, or a summary cloze about the evolution of a particular genre) would have yield more detailed information about the effect of our multimedia program on students’ comprehension skills. As assessed in this experiment, however, students’ comprehension skills do not appear to be affected by the use of CD dictation and multimedia.

E. Conclusion

This study represents an initial step in an effort to understand the role that music, informed by hypermedia, might play in fostering students’ language development. The study supports the hypothesis that the simultaneous presentation of spoken and written language through multimedia can substantially improve the listening skills of the language learner. Compared to traditional training, the new instructional method, including CD dictation and multimedia, offered more opportunities for student exposure to the target language. Simultaneously engaging the aural and visual receptors of students had a positive effect on students’ language output as expressed in the listening scores of the last four testing sessions of the study. Moreover, in the open-ended items of the questionnaire students reported that the exploration of the songs’ cultural context stimulated their interest in Hispanic popular music and led them to rely more on music for language learning. The large majority of students enjoyed using CD dictation and Ingenio and felt that their ability to understand the target language from a CD had dramatically improved. Several students reported residual learning or the involuntary rehearsal of a song long after the music stopped, which supports Wilcox’s claim.

Contrary to our expectations, students’ comprehension skills did not appear to be affected by the myriad of stimuli and data provided by multimedia. This could have been the result of a deficiency in the assessment instrument used to test comprehension, in terms of the elicitation techniques used to test the skill. Due to the small sample size and the manifestation of some confounding variables these results need further empirical assessment.

One of the most challenging aspects of this type of study is the handling of a class as an experimental unit, while avoiding detractions from the learning experience. Conversely, trying to execute an experiment across different classes or sessions using different teachers and materials in each raises issues of confounding variables such as: difference in teaching styles and staff, seasonal issues that affect performance and class composition. To deal with these difficulties CALL researchers recommend focusing more specifically on finding what components of CALL lessons are effective, with what kind of lesson and for what kind of students (Miech, Nave & Mosteller 78). This perspective, coupled with a rigorous experimental design that includes quantitative and qualitative analysis, can turn our empirical assessment efforts into a fruitful source of information. (Other published studies on assessment of multimedia instruction can be found on Appendix C).

V. Summary of recommendations by CALL researchers

• CALL researchers overwhelmingly recommend abandoning the CALL vs. non-CALL design and focusing more specifically on finding what components of CALL lessons are effective with what kind of lesson and for what kind of students (Miech, Nave & Mosteller 78).

• We are also invited to look at the components of CALL lesson design such as: error and feedback analysis, screen design, program branching, etc (Miech, Nave & Mosteller 79).

• Another recommendation is to take into account student characteristics (learning style, gender, cognitive approaches to learning) (Miech, Nave & Mosteller 79).

• CALL researchers also recommend that we agree on a research agenda or on key questions to be addressed in subsequent studies.

A. Questionnaire used in the Ingenio study

1. The utilization of multimedia technology was beneficial for the development of my language skills, that is:

a. Reading SA A N D SD

b. Writing SA A N D SD

c. Cultural understanding SA A N D SD

d. Oral ability SA A N D SD

e. Listening ability SA A N D SD

1. Learning the cultural context presented in our music program Ingenio improved my language skills.

SA A N D SD

2. CD dictation improved my ability to listen to, and understand a song.

SA A N D SD

 

 

3. Multimedia approach to language acquisition enhanced my learning of the language more than traditional approaches.

SA A N D SD

 

4. The use of technology on this course sparked my interest in this field of study (Spanish language, culture, and literature).

SA A N D SD

5. Please explain how using multimedia and web-based materials affected the development of your language skills.

 

6. What did you find most valuable about using multimedia in this course?

 

7. What did you find least valuable about using multimedia in this course.

B. Minute card (example)

The listening scores for a group of 25 students is shown below. An asterisk denotes missing scores (the students did not take the test for some reason) and their scores cannot be used to calculate the reliability coefficient because pairs of scores are needed for each student.