Dr. Rob Schoonen - "Generalizability, where theory and methodology meet"
Bionote Rob Schoonen is an associate professor at the Department of Linguistics and Literary Studies at the University of Amsterdam. His main research interests are second and foreign language acquisition, models of language proficiency and assessment, and research methodology and statistics in Applied Linguistics. He has published on several methodological and assessment issues in SLA, and he has served as associate editor of Language Learning. Abstract One of the primary goals of scientific research in SLA is to gain insights that will help us to make decisions on, or predictions about, language learning. An underlying assumption is that the data we have collected, allow us to make valid decisions and predictions, that is, decisions and predictions that are generalizable to new cases, other measurements, and maybe new situations. In current L2 research, and applied linguistic research in general, little attention is given to the generalizability of measurements and research outcomes. As a research community, we run the risk of building interpretations on incidental outcomes. In the first part of the presentation a number of these ‘generalizability’ problems will be outlined in more detail, such as sampling of participants, sampling of measures and also interpretation of statistical outcomes. In the second part, I will focus on measurement and show that psychometric issues are in principal theoretical issues. Finally, a few possible ways to deal with these generalizability issues will be discussed. Answers to questions about generalizability of research findings are needed in order to be able to build a solid and generalizable knowledge base in our field of SLA. |
Dr. Martin Lamb - "Researcher effects in qualitative inquiry"
Bionote Martin Lamb is a Senior Lecturer in TESOL at the School of Education, University of Leeds, UK. After qualifying as a teacher of History, his early career was spent overseas as an EFL teacher in schools and universities in Sweden, Indonesia, and Bulgaria. In Leeds since 1999, he now manages the MA TESOL programme, teaches undergraduate and doctoral students and researches in the area of language learner and teacher motivation. Abstract Experimenter/investigator/observer effects – and other variants on the original Hawthorne effect - are well-known in scientific research as potential threats to reliability and validity. In qualitative research they are not usually considered as threats since the reflexive researcher must always consider their own role in the research procedure, how they may be influencing the behaviour of participants and introducing bias into their analysis and interpretation of data. In this talk I will review these necessary precautions, but also consider the possibility that our qualitative studies have longer-term effects on our participants, perhaps years after the research has ended, and these may have ethical implications. |
Dr. Luke Plonsky - "Reliability Coefficients in L2 Research: A Meta-analysis"
Bionote Luke Plonsky (PhD, Michigan State University) is Assistant Professor of Applied Linguistics at Northern Arizona University, where he teaches courses in quantitative research methods, SLA, and L2 pedagogy. Recent and forthcoming publications in these areas can be found in journals such as Applied Linguistics, Language Learning, Modern Language Journal, and Studies in Second Language Acquisition and in several edited volumes. Luke is also Associated Editor of Studies in Second Language Acquisition and Managing Editor of Foreign Language Annals. Abstract Concerns have been raised recently over the lack of reliability estimates found in reports of published L2 research (e.g., Cohen & Macaro, 2010; Plonsky & Gass, 2011). A potentially greater threat to internal validity is the actual reliability of data collection instruments being employed. Unfortunately, L2 researchers are left to interpret the performance of their instruments, if they do so, in relation to generic benchmarks. In response to these issues, this paper meta-analyzes reliability estimates (internal consistency, interrater, intrarater) as reported in the field. We also explore heterogeneity in observed estimates as a function of different coefficients and of study and instrument features suggested to moderate them (e.g., Brown, 2014). More concretely, we recorded 2,244 reliability coefficients found in 532 published studies of L2 research along with a number of features corresponding to the designs (e.g., sample size, proficiency) and instruments (number of items, test formats) employed. We also coded for the type of reliability index reported (e.g., alpha, KR20). The three types of reliability varied, with the lowest estimates found for internal consistency: median = .82. Interrater and intrarater estimates were substantially higher at .92 and .95, respectively. These overall estimates were also found to vary according to several study and instrument features such as learner proficiency (low = .79, intermediate = .84, advanced = .89) and target skill (e.g., writing = .88 vs. listening = .77). In addition to raising L2 researchers’ awareness of the need to report reliability and other psychometric features (see Norris & Ortega, 2012), we use our results to inform and encourage interpretations of reliability relative to the larger field as well as to the substantive and methodological features particular to individual studies and subdomains. |