Short little mother

"You're a short little mother", he said

I read this phrase in a book, apparently someone told that to Yul Brynner. it sounds derogatory but what is the exact meaning of the phrase?

thank you!

Daniela