[fizlfry]

Before I read this forum, I took a very used 36 grit sander to the whole frame. While now I suggest not doing it, It sure took care of all the deep scratches, the anodize, and anything else might have been on there. So, if you are good at sanding, and know you wont gouge that edge into the surface, you could use like 80 to 100, and I bet it would take it down just fine. I used a high speed air sander.
About your swing arm.....Unless you plan on never riding this, and storing it for show, I would not worry too much about the tiny scratches...If you are anything like me and ride alot..You are bound to pick up road ****/rocks/etc...onto that swingarm. I would say just be careful, try to prevent as much as you can, but, your rides will be worse on it.
Hope this helps. I learned all I know from this forum.