I actually might be able to provide some insight to this. My wife is Balkan and the first time I called her a “Silly Goose” she became madly upset. It turns out in her language the phrase “Glupova Gusko” (Stupid Goose) is a common insult. It is considered incredibly harsh in her culture. My guess is that the phrase “Silly Goose” is borrowed from a Slavic Language and lost its harshness when it moved into English.
I knew about the honor-culture bit, but I’m now curious about what other languages use it as an insult.