No, they don’t have legitimate concerns. Refusing to debate them is not impinging on their right to free speech. "Cut their heads off and fill the holes with garlic, that's what you do with vampires." – Terry Pratchett