A Standardized Test with Unusual Results

LEXINGTON, Ky. (Sept. 21, 2010) − Recent University of Kentucky graduate Graham Drake continued the winning tradition of his Martin School of Public Policy and Administration peers this year with a National Association of Schools of Public Affairs and Administration (NASPAA) Best Master’s Student Paper Award, and the paper, titled, "Evaluating the Presidential Management Fellows Program: Has the Implementation of a Standardized Assessment Test Altered Student Selection?" all began with one long drive.

Last year, Drake traveled from Lexington to Chicago to take the Presidential Management Program exam, only to have the identical test administered electronically. "We got back to Lexington the night before the first day of classes," he said. "And I wasn't alone. Most people taking the exam were from Ohio, Indiana and Kentucky and had driven at least two hours to take a test they could have taken from home."

The federal government's Presidential Management Fellows (PMF) program was in need of revision. Drake's internship last summer with NASPAA provided the last push he needed: after only days on the job, he learned that one of the association's major tasks for the summer would be "re-imagining federal hiring programs, including PMF."

The PMF program began in 1977 as a way of jump-starting talented graduate students into federal service careers. Finalists are selected from top graduate schools and then given the opportunity to connect with various federal agencies. If a finalist is hired by an agency, they are fast-tracked into that agency’s managerial ranks.

A few years ago, the federal government instituted a new system for choosing PMF finalists; applicants were chosen solely on the basis of a three-part standardized test, instead of face-to-face interviews, which the program had previously implemented.

"I felt that the skills test was hampering student selection," Drake explained. "There were definitely some differences from what they'd done in the past."

Drake's assumptions were on the right track. NASPAA had been hearing rumblings that the best students were no longer being chosen for federal jobs. "It was a hearsay thing, but they weren’t happy – the professors, deans and program directors – because they felt like the best students weren’t being selected," he explained.

Since the beginning of the PMF program, public policy grads had always accounted for a large number of the finalists, and in the wake of the new application process, NASPAA had been receiving negative feedback.

"Yes - this was a cost-cutting measure," added Drake, "but is it really the best idea in order to get the people that you want?" Drake wanted to know – had the new application process changed the makeup of the PMF pool of finalists?

So, he submitted a request under the Freedom of Information Act asking for data on the PMF applicants and finalists from 1997 to the present. According to his results, the characteristics of the finalists did change along with the application process. Specifically, fewer public policy and administration students – and an increasing number of law students - were becoming finalists.

Drake emphasized that his research was not able to determine any changes in the quality of those PMF finalists who became federal employees, because the Office of Personnel Management did not provide him with data about the fellows – that is, those finalists who were actually hired by a federal agency. But he does think his results show there is a need for further examination and possible change.

And it seems the federal government agrees. "Since I wrote my paper – not because of the paper – they’ve re-instituted the interview process after the test," he said. "Standardized tests are helpful, but an interview is equally important, and I think that there is room for both.

"Graham's paper is one piece of information NASPAA can use as they try to shape and inform the federal selection process," said Edward Jennings, one of the many Martin School faculty members that assisted Drake with his research along the way.

Jennings also helped The Martin School graduate submit his paper to The Journal of Public Policy & Education and feels confident that it will be accepted, promoting Drake's research to a wider audience.

Drake is proud of the NASPAA award, but attributes his success to the Martin School faculty. "Nine-tenths of the skills and statistical tools that I used in that paper I didn’t have before," he said. "It's nice to live up to the Martin School reputation."

Check out Drake's interview online at http://www.martin.uky.edu/news_drake.html.