i think what irks me about "it's fine to x, but" phrases is that it's implying that you should reflexively forgive someone for the irritation of doing x, but then you just skip that part cause you want to complain. so just complain, stop qualifying your speech