4 Cabinet members have left this term, and they all have 1 thing in common.