Topic Thread

Topic: AAPPL Data Talk

1.  AAPPL Data Talk

Posted 30 days ago
Dear Language Educators Community,

My name is Fatima Baroudi and I am an Arabic teacher and World Language Department Coordinator at my school American Cooperative School of Tunis. For assessment we have been using AAPPL test as an external assessment for 4 years now. Even though this test is a performance test, we use as an external assessment to all students ( heritage and non-heritage) except for the IB students. We never prepare the students for this test and we don't necessarily follow the same topics that are covered in the tests, so in a way we use it as a proficiency test. I was wondering who among you use this test. Who do you give it to? How often do you give to? How do you use the collected data? Do you go over the students writing and oral performance? Do you find the scores go with the score descriptors? All of these are questions that we always ask in my department and we would love to hear from other schools' experiences. Shukran Jazeelan!

------------------------------
Fatima Baroudi
fbaroudi@acst.net
------------------------------


2.  RE: AAPPL Data Talk

Posted 29 days ago
Good evening,

My name is Gisela Cordero-Cinko and I am a Spanish teacher in the state of New Jersey. I am not familiar with the test you mentioned but I am familiar with the STAMP exam, which is similar to the one you mentioned. We also ask our students to take the test without preparing them in order to assess their proficiency level, we use it as our benchmark in their junior year. This allows us to see where our students are and which areas we need to work on in their last year of studying a language as seniors for those who choose to continue. 

The test does allow us to see what students say and what they wrote in their speaking and writing sections, but we do not discuss their answers with them. Not this year anyway. We will be using the good ones (the ones where students reached Intermediate Mid or higher) to show our students next year want a 4 answer is vs a 5 or a 6 (4 is Intermediate Low, 5 Intermediate Mid and 6 Intermediate High). This will help them see that even with errors they can achieve the IM level. 

I think it is best if your school continues to let the students take it without prepping them. This shows truly what their abilities are in the language they are learning. However, as a department you should use the results to discuss ways to increase and improve in the areas where students perform poorly. For example, we now know we need to use more authentic audio and videos with background noise because our students did not perform as well as we expected them. 

I hope this helps.


------------------------------
Gisela Cordero-Cinko
gcorderocinko@gmail.com
------------------------------



3.  RE: AAPPL Data Talk

Posted 29 days ago
Thanks, Gisela! I appreciate your prompt response. Yes, we have been using this test for 4 years now and this year we had it earlier so we have time for talks and discussions. I have just googled the STAMP and I think it looks very much like the AAPPL. I like the way you are using to help students move along the continuum. We are still receiving results for this post-test this spring window, and I can tell that the students have made growth. However, I can tell there are some surprises when I click on students written and oral answers. Sometimes the scores are higher or lower than I would give. I know I am not an expert but when I look at the score descriptions and the scores I get confused. I am sure I am missing something. For instance, I have  fluent french speakers who wrote in a very accurate language using connectors and details got an I4 ( High Intermediate), while other students who wrote with so many errors got an A ( Advanced). I am still not very clear on this and I would love to hear your input and from other language teachers who are familiar with AAPPL. Thank you so much for your responses! 

------------------------------
Fatima Baroudi
fbaroudi@acst.net
------------------------------



4.  RE: AAPPL Data Talk

Posted 28 days ago
Fatima-

I am so happy that you made the comment about the surprise you have sometimes when you listen to or read what your students have submitted.  I too have felt more than once that my students received a higher or lower grade than I would have given them.  I am not an expert either, but went through an OPI week long training a few years ago and feel like I have an idea of the difference between an Intermediate and an Advanced ranking.  I am wondering if there are other teachers out there that feel the same way.  

------------------------------
Kathryn Stoyak
kstoyak@lchsrailers.org
------------------------------



5.  RE: AAPPL Data Talk

Posted 26 days ago
I am interested in this statement, "I have  fluent french speakers who wrote in a very accurate language using connectors and details got an I4 ( High Intermediate),".  Based on what assessment are you measuring them as 'fluent french speakers'?  Lastly, what is your definition of 'accurate language'? I'm wondering if these measurements, fluent and accurate, are tied to a certain set of standards? This might help us get to the answer, as to why there are differences in scores.  Thank you.

------------------------------
Randy Barrette
Instructor, Morehead State University
r.barrette@moreheadstate.edu
------------------------------



6.  RE: AAPPL Data Talk

Posted 26 days ago
Thanks for your response! By accurate I mean grammatically correct, including the use of subjunctive, imperfect, past and present tenses with different pronouns, the use of complex sentences, etc. I thought that was called accuracy and for fluency I meant the long and descriptive sentences, idioms, etc. My assessment is my observations during authentic tasks at school such as interaction with the local communities in field trips, etc. For the set of standards, I am not sure, I think I need to go back to the standards and look carefully whether or not grammar accuracy and fluency are mentioned somewhere. So, do you think as long as the students are functioning authentically and communicating their language accuracy should not be a problem? If so, then may be this is why there are differences in socres. 

------------------------------
Fatima Baroudi
fbaroudi@acst.net
------------------------------



7.  RE: AAPPL Data Talk

Posted 28 days ago
Gisela,

I am wondering if you, like Fatima and I, see a disconnect at times between student performance on the STAMP exam and their rankings?  I would appreciate hearing your thoughts.

Thank you!

------------------------------
Kathryn Stoyak
kstoyak@lchsrailers.org
------------------------------



8.  RE: AAPPL Data Talk

Posted 28 days ago
Thanks Kathryn for your response ! What a relief when you see you are not alone in your findings and observations. By the way I just found out that if you disagree with the ratings or scores you can request a diagnostic feedback report from the customer care, which I did and am waiting for a response. I have another question for you and those who use AAPPL in their schools, Other than the rating inconsistencies do you have issues with the questions/ prompts? We give this test to both Arabic and French students and the problem here for students who take both tests, since the tests are identical, the test which is administered second loses its validity since the students know already the content. Am I making sense here? Thanks everyone for the very valuable feedback.

Sent from my iPhone




9.  RE: AAPPL Data Talk

Posted 27 days ago
I gave the AAPPL assessment for the first time last year to my Spanish stuents.  I also found some discrepancy between performance on this assessment and my evaluation of some students´ proficiency level, but overall, it was close enough that I found it a valuable exercise. I plan to use either the AAPPL or the STAMP test this year. One benefit of the AAPPL is the price--just $5 for the Reading and Listening components, compared to about triple that for the STAMP.

I wrote up a lengthy document about the experience that you may find helpful. The slide show I used to debrief the testing experience with my students is on the sidebar. (If you do use a test like this, I think it is very important to help students understand what their ranking means... or doesn´t mean. It can be a helpful exercise, but should not be taken too seriously, in my opinion.)   http://spanishfromtheheart.weebly.com/for-teachers

------------------------------
Rita Barrett
rbarrett@paasda.org
------------------------------



10.  RE: AAPPL Data Talk

Posted 27 days ago
Thank you so much, Rita!

You inspired me to come up with something about our experience with AAPPL. I completely understand what you felt and I do share most of your thoughts. We are still finishing so the final data is not collected yet. As I might have mentioned in my previous thread, at our school, American Cooperative School of Tunis, we administer this test once a year and for new students twice a year ( fall and spring). All students taking French and Arabic are given the AAPPL test, a lot of who are heritage speakers and they do rate as high as A and may be more if the test had something higher but it does not. I think what we will do we will print out the students's work ( BTW is there a way we can save the audio files of their speaking?) and will give them the descriptors to see how they would evaluate themselves. Looking at the number of hours is actually a nice guiding principle so they realize what they achieved or not during a specific time frame. I will let you know what we did as a follow-up I appreciate your feedback, Rita! 
Have a nice week!

------------------------------
Fatima Baroudi
fbaroudi@acst.net
------------------------------



11.  RE: AAPPL Data Talk

Posted 26 days ago

Great discussion of assessment issues in the ACTFL Online Community.  We would like to share a few observations as well:

  • The AAPPL assessment contrasts with a classroom-based assessment because it’s developed and rated externally.  One reason we rely on external assessments (such as STAMP or AAPPL) for certain kinds of assessment rating and even program evaluation is that they will be more objective about student performance, basing the evaluation only on the evidence presented (what the learner can do in the sample provided) and not on any prior classroom performance.
  • Classroom learning checks and classroom-based summative performance assessments are based on what is taught (and how it is taught), frequently allowing learners to correct, rehearse, and revise.  Classroom evidence is more practiced and coached – and learners frequently rise to higher levels of performance in that setting.  When we were teachers, we often reflected on the best of what our students produced and felt they were at the “Advanced” level, but then realized that those five-paragraph essays took three weeks to produce, with a lot of editing and feedback!  In contrast, an outside assessment like AAPPL or STAMP provides unrehearsed, uncoached, and “on demand” evidence of what students can do
  • It is important to keep in mind that no single response is sufficient to evaluate a proficiency level.  A rating needs to consider what language is sustainable, what patterns of errors exist, and what level of control is demonstrated. AAPPL raters don’t just listen to or read one response and rate it, but rate as a whole an entire sample on one topic and identify if the learners can produce language at the targeted level (and if they can, whether the evidence is strong or minimal of that targeted level).  The test taker is given multiple opportunities to produce language at the targeted level, so the overall rating is not dependent on one well-rehearsed topic.  In contrast, in a classroom, we spend many weeks on one topic – in a unit of instruction.
  • Mistakes need to be analyzed carefully to determine their impact on a rating of one’s proficiency.  A rating of proficiency is not about counting mistakes but about identifying what the language user controls.  That is what is described in the Proficiency Guidelines (and their translation into learning objectives with the NCSSFL-ACTFL Can-Do Statements).  Learners with less language have fewer opportunities to make mistakes.

Meg Malone, Director, ACTFL Center for Assessment, Research, and Development
Paul Sandrock, ACTFL Director of Education

 



------------------------------
Paul Sandrock
ACTFL, Director of Education
psandrock@actfl.org
------------------------------



12.  RE: AAPPL Data Talk

Posted 20 days ago
Edited by Gisela Cordero-Cinko 20 days ago
Sorry I took so long to respond. I think what has helped me understand the reasons behind these "discrepancies" was to go through the OPI training a few years ago. Going through that training not only helped me understand the differences between the different levels of proficiency but it also helped me be a better assessor of my students abilities. By that I mean that I learned not to be what they call a "sympathetic listener" to my students. When I participated in an AP training class, I was very happy to learn they focused more on students completing the task, in order words answering the question accurately, than the grammar.

I guess what I am saying is: take a look at the question, read the answer and then ask yourself did my student complete the task? Did he/she answer the question? How much detail did he/she have? Was it organized? Would a native speaker understand their answer with ease?

I am usually in agreement with their evaluation for the most part. I do not think I helped in any way. I hope I did a little.

Have a good week....

PS: Read Paul's response. He explained it much better than me.

------------------------------
Gisela Cordero-Cinko
gcorderocinko@gmail.com
------------------------------



13.  RE: AAPPL Data Talk

Posted 20 days ago
I have received training in ACTFL OPI rating, so when I talk about discrepancies, I am not speaking as the teacher who hoped students would rate higher because they do well on class performance. The discrepancies that caught my attention were a student who is fluently conversant at an Intermediate High/Advanced Low level receiving one Novice High rating; a student scoring I3 for writing, but I1 for reading; and a student who is definitely in the novice range who received Intermediate scores. A few just seemed high or low one or two steps.

Many students complained that ambiguous visuals, rather than language difficulty, caused confusion and there were technical glitches on our side that could have caused some testing problems. However, looking at my students overall, ratings matched reality enough to make me want to use the AAPPL again. I do think it is very important to tell students that this is one test on one day and not an absolute determination of their actual proficiency. For me the most important reason to use the AAPPL or any other proficiency test, is to encourage language learners to think about proficiency, rather than grades, and it was generally a positive, encouraging experience for my students.

------------------------------
Rita Barrett
rbarrett@paasda.org
------------------------------



14.  RE: AAPPL Data Talk

Posted 12 days ago
Good afternoon,

I re read all of the responses on this topic. I think we see discrepancies on these tests based on how we assess or are forced to assess our students (in my case it is forced since I do not agree with students filling bubbles on any tests). I do have freedom when I assess them in class as part of a unit (which I usually create using backward design and ignore number of pages) but lose it when we give them their midterm & final exams.

What helped my students to be able to complete the STAMP, is that they felt some of the tasks they had to complete looked similar to tasks I asked them complete in class (the ones I used to assess proficiency and not performance). I have given this test before, so I tried to include activities similar to what they were going to see on test day. That does not mean they perform with the scores we were hoping for level 3 students in every section, but for the most part the results were good. We gave this test for the first time to award the Seal to our students in their senior year. That was and is our goal....

We were not upset with the results, but we know now what we need to work on for next year. By that I mean, we know what we need to tell our teachers at the lower levels and know what we need to do at the higher ones. We have a good benchmark.

There should be a way to assess our students where we do not give a grade/percentage but a good/accurate assessment of where they are in the language in our classrooms. If in the future we can develop a way to do this for WL classrooms, we will be better equipped to get our students ready for 21st Century. It is a dream....

------------------------------
Gisela Cordero-Cinko
gcorderocinko@gmail.com
------------------------------



15.  RE: AAPPL Data Talk

Posted 12 days ago
"There should be a way to assess our students where we do not give a grade/percentage but a good/accurate assessment of where they are in the language in our classrooms."

You're right on, Gisela!  I believe that is what is behind the whole push for implementing ACTFL proficiency levels in our classrooms, and moving students along those proficiency levels as our guide - which has not been tradition.  I think that this new emphasis will be transformative as it catches on.

I think that a major hang-up that many teachers are having - I am one of them - is what happens to *grades* under a proficiency-based model of teaching.  I don't have an answer for that, and I haven't heard a good one yet.  Weigh in, ACTFL community!  Perhaps the recent issue of "The Language Educator" will address this more; I've only read one article from it so far.

On one hand, I hear people say that a first year course should get students to Novice-High.  The logical conclusion would be that you give an assessment at the end of the year with Novice-High tasks that would determine whether or not students have reached that mark.  But on the other hand, I am hearing a lot of "But every student moves at a different pace; students acquire languages at different speeds, so don't penalize them if they haven't reached a certain proficiency level by X date".  So while I appreciate the new proficiency paradigm, and I've started to take a serious look at my scope & sequence in the light of it (i.e., Can third-year high school students be arguing for environmental policy in their first semester?  According to Avancemos, yes; according to 'proficiency', probably not), I don't know how to reconcile these competing voices.


------------------------------
Chris Cashman
ccashman@chiarts.org
Spanish Teacher
------------------------------



16.  RE: AAPPL Data Talk

Posted 11 days ago
When you say "You should get your students to Novice High" by the end of the first year, do you mean after one year of high school or college language instruction?

Also, the teacher does not "get" their students to any certain level. The students have to do the work. They have to "get" themselves there!

Emily

------------------------------
Emily Serafa Manschot
vayaazul@aol.com
------------------------------



17.  RE: AAPPL Data Talk

Posted 10 days ago
Hi Chris, I think there are exit criteria, performance descriptors or proficiency ones, depending on what our goals are. However, there are students who reach these exit criteria slower or faster than others, like in other disciplines. We have been using the AAPPL for 4 years now and I think there are always surprises in our data. Being the tests coordinator at my school, I have access to the entire data and I can see that there are often students who have been studying the language for more than 3 years and they are still at N4 or students who are in their first year and they are I2 or I3 across the different skills. I am not sure how to explain this other than the fact that the students have a faster pace than expected. Any thoughts on this?

------------------------------
Fatima Baroudi
fbaroudi@acst.net
------------------------------



18.  RE: AAPPL Data Talk

Posted 10 days ago
Hi, all  -

Interesting responses so far.

I should have clarified that I was referring to a high school first year course.  And in reality, I was just using "Novice-High" as an example.  The larger idea is that I see a new movement to attach proficiency-level (goals?) to language courses - which I think is a wise thing, but doesn't come w/o some questions.  I agree that students have to get themselves to a required proficiency level.  The question is, if they don't, do they repeat the class until they hit it?

For example...

Passing first year = achieving Novice-4, based on some measure (like AAPPL, a certain type of OPI w/ teacher, etc)?
Passing second year = Intermediate-1, based on some measure?




------------------------------
Chris Cashman
ccashman@chiarts.org
Spanish Teacher
------------------------------



19.  RE: AAPPL Data Talk

Posted 10 days ago
According to my humble experience as an Arabic teacher, there is no way you can reach I2 in the second year of instruction. Well, sometimes there are exceptions, but generally speaking N4 is not reached after one year of instruction. When I say one year meaning, average of 3 hours a week, 12 a month, and let's say 100 hours per 9 months. At least this is how it is in my school. Then, the intermediate some students might take up to 4 years in this level with all of its sub levels. How do others who teach non roman languages feel about this?

------------------------------
Fatima Baroudi
fbaroudi@acst.net
------------------------------



20.  RE: AAPPL Data Talk

Posted 10 days ago
On "doing the work..."

Actually, there are so many internal constraints that students can only "do" so much. Same goes with the teacher's role. I would say that the teacher should "do" their best to make language comprehensible for the student. What the student can "do" is indicate when something isn't comprehensible, which can certainly help. After that, the student's internal syllabus takes over. There will be 4th year high school Novice students, for sure.

If schools want us to assign a grade for the process of acquisition that we have limited control over, most students should be getting an A(95) no matter what their rate of acquisition is.

------------------------------
Lance Piantaggini
magisterp.com (Teaching for Acquisition-Making Latin & other languages More Comprehensible)
lpvisual.com (Marching Artistry)
------------------------------



21.  RE: AAPPL Data Talk

Posted 11 days ago
Maybe it is time we ask the students what they think about proficiency levels and where they want to be or think they ought to be?  What does the student want to do with the language and what level of proficiency would work for him or her?  Do you want to work or travel in that country--what level of proficiency do you need to accomplish that?  Do you want to study abroad? If so, what level do you need?  If we put the goal onto the student, then they may feel motivated to reach that level and to choose the appropriate types of instruction to reach that goal.  From there, maybe we need to create courses that address the different needs and types of proficiency rather than one method for all.  The final test and grade assess whether the student has reached his or hew own proficiency goal.

I'm retired, so you don't have to take my ideas seriously--but maybe just as a discussion point?

Suzanne C. Joy, retired teacher of German

------------------------------
Suzanne Joy
Kennebec Valley Community College
looney@uninets.net
------------------------------



22.  RE: AAPPL Data Talk

Posted 10 days ago
I like Bill Van Patten's solution: just give a proficiency rating at the end of the year, no grade. But I don't know how that would work in our GPA-oriented high schools.

------------------------------
Rita Barrett
rbarrett@paasda.org
------------------------------



23.  RE: AAPPL Data Talk

Posted 9 days ago
IMO, the ACTFL proficiency scale is not fine-grained enough to be of much use for giving grades or even progress indicators during a typical K-12 program. Certainly not year-to-year, even for high school.

That's really not what it was intended to do, though, was it? It seems like it's aimed at language throughout the life cycle, really.

Before adopting the ACTFL proficiency scale as the basis, we also need additional attention to the question of students taught through comprehensible input (who "acquired" rather than "learned" the language) as their performance indicators do not seem to jive well with what's listed as the standard progression as per the ACTFL standards. I believe that scale was based on studying languages. There's a lot more going on out there now given the explosion of CI methods and also Internet based resources that provide non-traditional instruction or exposure to languages.

------------------------------
Terry Waltz, Ph.D.
SquidForBrains Educational Publishing
Mandarin through Comprehensible Input
------------------------------



24.  RE: AAPPL Data Talk

Posted 10 days ago
That's a great idea, but it poses a problem.  How do colleges/universities distinguish who should be admitted if grades are replaced with proficiency levels?  Also, we have some problems with the way the AP course it outlined.  When students finish Level 4 they should be at a certain level but then expected to bring themselves up to an even higher level on their own by the time they start AP.  At our school it's double hard as we don't have a Level 4 so it goes right from Level 3 to AP.

------------------------------
Tom Beeman
High School Spanish Teacher
California Virtual Academies
------------------------------



25.  RE: AAPPL Data Talk

Posted 28 days ago
Fatima,

As a follow up to my first response, here are my answers to some of your other questions.  Our purpose of giving the test is a little different from yours.  We administer the tests to Spanish 3 and 4 students who choose to take it.  They are taking it to earn the Seal of Biliteracy and according to Illinois law they must earn an I-5 or higher on each of the sections.  Up until this year, students have had the opportunity to take the demo test and go over it with me. This year students who choose to are actually practicing for the actual topics that will be on this year's AAPPL writing and speaking sections.  The students' results are used as the basis to award the Seal.  

------------------------------
Kathryn Stoyak
kstoyak@lchsrailers.org
------------------------------