I'm not saying I'd wear it all the time. But if I'm buying a LBE chest rig then it might as well be able to carry plates or soft armor.
At my age the weight isn't an issue. In in a long distance but out situation you can all ways ditch the plates if it was an issue.
You should wear it all the time. Train like you fight and fight like you train. Your plate carrier should have your mags attached directly to it. If you want a light weight set up - build a Battle Belt with (3) rifle + (2) hg mags. The belt can be used in conjunction with the plate carrier.
Weight is always an issue. This is regardless of age and physical conditioning. More weight on other crap = less ammo and water. More ammo and water = longer life span.
I'm not preaching at you - I've just been doing this for 30 years and have a few hard earned lessons to share.