You MUST keep the wheels clean. As well as getting gummed up with compound, used buffs contain fine pieces of the metal you are polishing and will leave scratches in your part.
The 6" buff probably didn't do as good a job because your RPM's are too low. A 4" buff will spin faster than a 6" buff, therefore working more efficiently.
Read our How to Buff and Polish Booklet at
http://www.caswellplating.com/buffman.htm