I'm not even sure he couldn't, honestly. He describes hobbits mainly via their personalities, having curly hair and brown skin and big feet. We the audience aren't required to interpret that as "white" just because of when he was born or where he lived. Neither of which were all white, either.
