People don't know their neighbors as well, and being exaggeratedly cautious about letting your kids out of your direct supervision is a signal for responsible parenting in many social circles (see: sleepovers).