Indeed, the comparison is quite loose. Not impossible to compare, but silly to do so. There are many things, accidental to human nature, which are not a choice. Height, would be an example. Odd number of hairs also. And yes, these are not just like or properly comparable to discrimination based on color.