If you want to blend in and not look like a threatening target then you are going to need to dress like a normal Joe off the street.
To me it's not so much the color as the military look. You take a black backpack that is just a normal civilian book/hiking backpack and it won't raise any eyebrows. My Camelback HAWG (civilian version) is black and looks very normal. I have a Camelback Alpine too and it's black. <img src="/ubbthreads/images/graemlins/laugh.gif" alt="" />
It's not so much the color, but the combination of color with style, your other gear, and how you carry yourself. You take a three day assault pack with MOLLE webbing and you have much more tactical look going on. You take that same pack and add a big honkin' knife strapped onto the side and you are going to get much more attention than if you look like you are headed out for a dayhike with a "normal" pack and your knife tucked away.
I don't like all of that attention, that's all I'm sayin'. <img src="/ubbthreads/images/graemlins/smile.gif" alt="" />