Just a bit of language policing; ‘kid gloves’ does not mean gloves worn by a human child, but rather gloves made from the skin of baby goats, known for their softness and suppleness.
Where the idiom comes from is lost in a bottle of whiskey somewhere, I’m sure.