I've had fantastic luck using the Hoppe's Rust and Lead removal cloths:

http://www.midwayusa.com/eproductpage.exe/showproduct?saleitemid=210734

They are made for cleaning guns, but are just as useful on knives. They usually go for less than $4.

And for prevention in the future, you should think about picking up a Tuf-Cloth. I've never seen anything rust that was treated with one of those before storage.