I tell pilot candidates I train that if they score 100% on their knowledge test, I will buy them a margarita, assuming they are of legal drinking age, or a Peet's cappuccino if they aren't. You see the current FAA's knowledge testing procedures have all the characteristics of a game. I'm not being flippant, I'm being honest. The tests are expensive to take ($150 on average), a bank of representative questions for each test is published in advance, you're only offered a quick review of any questions you missed, you won't be told the correct answers to any questions you missed, and the tests have historically contained trick questions and misleading graphics. And there's a whole test preparation industry that produces books, weekend study courses, and computer-based training all designed to help applicants get a good score. The only thing missing is popcorn and liquid refreshment.
All was generally well with the FAA knowledge tests until, without any notice, the FAA decided to change the rules of the game by changing the test question bank for the Airline Transport Pilot, Flight Engineer, and the Fundamental of Instruction knowledge tests without modifying their own publications. The predictable result was a sharp increase in the failure rate and that has led many to observe with shock and horror that what had heretofore been occurring was not learning at all! To better understand the situation, you'll need to set aside some popularly held misconceptions about the FAA's knowledge test procedures.
Learning is ...
The instructional theory and concepts that flight and ground instructors are taught can be found in the FAA's Aviation Instructor's Handbook. I'll be blunt and say that the handbook contains a hodgepodge of educational theory, some of it germane and useful, some of it ... not so much. It has been edited and changed over the years and in the process has been somewhat improved. One of the core concepts presented is the four levels of learning: Rote, Understanding, Application, and Correlation. Rote knowledge is the memorization of facts while correlative knowledge is the ability to combine new experiences and information with what you've learned in the past. Most FAA practical test standards contain this boilerplate (emphasis added):
Examiner’s shall test to the greatest extent practicable the applicant’s correlative abilities rather than mere rote enumeration of facts throughout the practical test.
While a practical test is dynamic and interactive affair, the knowledge test is anything but. An examiner can ask questions selectively and in a sequence so as to uncover the level of knowledge the applicant possesses. The knowledge test is static (okay, the groups of questions are randomly selected at the time your test is generated) and a much more difficult affair since the test must provide valid results across a diverse population. It's good to have goals with regard to testing and examining pilots, but the truth seems to be that both the knowledge test and the oral portion of the practical tend to rely heavily on rote knowledge for the simple reason that it is easiest to test. I know one flight instructor candidate who was pink-slipped because he could not recite, verbatim, the definition of instructional level of knowledge.
Multiple Choice
A knowledge test should measure what the candidate has learned in an objective and effective manner. The easiest type of assessment to administer and grade is a selection style (aka multiple choice) test. Selection questions must be designed so that there is only one correct answer or one best answer. The Aviation Instructor's Handbook provides guidance to instructors on how to avoid using puzzle, trick, or bewilderment questions. Apparently in an effort to discriminate between levels of learning, the FAA often resorts to the very types of questions they caution against.
Some of the more egregious examples that come to mind include the graphics used in the Instrument Rating knowledge test where an RMI needle was depicted as slightly bent or the arrow head of the needle was depicted so subtly that one might mistake it for the tail of the needle. Then there were the flight planning questions where the en route time or fuel burn or magnetic heading that you calculated didn't exactly match any of the answers, but was somewhat close to one of the supplied answers. What was actually being testing here? No one seems to know, but a whole industry began to grow up around these tests.
In the past, when you missed a question on a knowledge test the only feedback your test results showed was the knowledge codes indicating the general subject area for incorrectly answered questions. The FAA changed this system somewhat and now publishes the Learning Statement Reference Guide for Airman Knowledge Testing. How they arrived at the term "learning statement" is beyond me, but I'm sure it involved hours, if not weeks, of meetings.
Preparation, Memorization, Commercialization
When suggesting to a student how to best prepare for the knowledge test, I ask them a bit about their individual learning style. Some people like doing computer-based training, others prefer a printed study guide, while others prefer the group learning environment provided by a weekend seminar. One size does not fit all. The advantage of a computer-based study system is that you can take sample tests in a format that closely resembles what you'll see at the testing center and that helps to reduce test anxiety for most people. I know of no students who receive their knowledge test preparation from their individual instructor one-on-one because the cost would be prohibitive.
Ideally, test preparation should help the candidate ensure that they are adequately prepared to pass the test and uncover areas they need to work on. What often happened in the past was that some candidates tried to memorize the answers and the tests end up measuring rote knowledge instead of correlative knowledge. To combat this, the FAA made some changes. First, the actual test questions are no longer published. Instead, a bank of representative questions is available for each knowledge test. Many people mistakenly believe that all the actual test questions are published in advance, but this has not been the case for several years.
The National Association of Flight Instructors has raised some important questions about the FAA's unannounced test changes, pointing out that some questions may have been added without first being validated. It also seems that the FAA reference materials no longer adequately represent the knowledge tests that applicants are taking and some subject matter areas may now carry more weight than they did in the past. As an aside, NAFI seems to be doing a commendable job of representing the interests of flight instructors and their students.
Cost of Learning
For quite some time now, the FAA knowledge tests have been computed-based and administered through testing centers affiliated with LaserGrade (now PSI) and CATS. The testing centers must adhere to proctoring rules, including having a closed-circuit surveillance system installed in the testing area. While there is a certain overhead to providing these facilities, it seems hard to justify the 100% increase testing fees that has occurred over the last few years. No one seems to have talked about this increase, probably because the test-takers are a captive audience and have no recourse but to pay the piper. It baffles me why some keep saying over and over that cost is not a significant impediment to learning to fly. With avgas pushing $6/gallon in many areas and knowledge tests costing $150, cost most certainly is a factor.
The FAA has tried to create a valid and reliable knowledge test for each rating or certificate and though the whole thing may seem to be a bit of a mess, historically, the FAA's testing set-up appears to be adequate. Applicants must expend some amount of effort studying and even if they are just memorizing answers, some learning is bound to occur during this activity. Secondly, the knowledge test is just one part of the learning equation. The last line of defense in ensuring that adequate learning has occurred is the oral portion of the practical test, where the applicant must stand and deliver. While problems are occasionally reported with the manner in which some examiners have administered practical tests, by and large the system, as a whole, works. So while aviation testing may appear to be a mess, it has been a mess that everyone understood and accepted. At least we thought we understood.
4 comments:
The trick to getting the correct answer on the time and distance questions to realize that the maps are not to scale. When the printed them they reduced them slightly. You can check this with your plotter. Measure the distance between latitude grid lines. It should be slightly over 60 nm. They never are.
When taking the test correct your measurements and you'll get the exact answer.
As an example, in one question on the Private Test you are asked to estimate the time from Mercer County Regional Airport (area 3) to Minot International (area 1). If you use the map without correction you will not find any of the answers, Using your plotter measure the tic marks in a line of longitude and see if they line up with the plotter. These do not. On this map 40 nm reads as 45nm, so our distances and angles will be off.
Now measure the distance between the centers of the two airports. It is 66.5 nm. Let's fix this by using the 40/45 as a correction factor. 40/45*66.5 = 60 nm.
You'll get the correct answer for all of the questions if you do this.
John,
Thanks for pointing out the scaling issue with the maps on the computer screen. That's useful to know if you are a pilot candidate about to take an FAA knowledge test.
My personal experience with FAA knowledge test answers that don't quite "add up" actually dates back to the days when a printed, paper guide was provided. The one problem I recall was strictly a word problem and no measuring was required.
I've not taken any of the newer computer-based knowledge tests that are "self-contained" with figures displayed on the screen.
Hah! Great post. As a CFI candidate I've now taken the PP, Instrument, Commercial, Flight Instructor Airplane, Advanced Ground Instructor, and Fundamental of Instruction written tests. And the worst, by far, are the performance charts. These charts and have reproduced since early IBM selectrics. Only the bible has been more copied. After following the peaks and valleys, headwinds and tailwinds, weight and temperature, does one arrive at 1,450 feet of takeoff distance. Answers: A) 1475 B) 1425.
Also, my flight school is a CATS testing center just as a service to the actual flight training: you would not believe the auditing and required ongoing training required by CATS. Essentially, it is so odious I expect that the testing centers to disappear and actually finding a testing center to be even harder then it is now.
I have no idea about how they line up on the screen. I'm referring to the printed booklets. Every single one of the sectionals on the Private Pilot,, and most of the sectionals on the Commercial test are not to scale. Take a plotter and measure 15 seconds latitude on a sectional. It comes out to exactly 30 nm. Now do the same thing on the test booklet. It is slightly less than 30 nm. That's true for every one of the sectionals in the booklet Private Pilot booklet. If you correct for the measurement error, or use a real sectional you get the exact same answer that the FAA did.
I can't imagine how you'd even attempt to do any measurements on the screen.
PS I was going from memory in my previous response and I wrote 60 nm instead of 30.
Post a Comment