There is one drawback of removing those guards. They act as a vibration damper. They are soft and heavy to dissipate noise emitted by those big body panels and help reducing road noise. When removed, you get the impression of closing doors of a little cheap car because of the "metal sheet noise".
Now mine are stuffed with Dynamat and it was planned anyway. But each time I get into my friend's car (6i without door guards), ouch, I don't like this noise. Sounds like a Swift door.