After reading the comment about Flitze, I remembered I had some Brasso Metal polish on hand. I worked on it a little with the polish and a paper towel for maybe 5 minutes. It seems to looks quite a bit better already. I will have to check it in the sun (its 1am) and see if there was a true improvement.

I do wonder about the dents through, any idea if INFI would have fared any better in this situation? My S5 Le got a dent but it was pretty easily steeled out.