I’ve noticed this as well, where I can’t listen to something with words (podcast, someone talking, TV show, etc) while doing a crossword or other word puzzle. I never really broke it down to realize that it’s likely because the part of my brain that processes language is completely incapable of multitasking.

I also have to avoid music with words while working; I’ll end up typing out the lyrics to the song instead of the email response to a user that I’m supposed to be composing.