There is a lot you can say without saying anything, especially in Japanese. But body language, like common sense, is culturally-determined, so learning a new idiom also involves adding new gestures, facial expressions and semi-verbal sounds to your repertoire. This unspoken vocabulary can be very helpful if you want to say something, but lack the motivation (or skill) to combine foreign grammar and vocabulary into a coherent sentence.
Interjectory sounds are picked up almost subconsciously; one day you realize you’ve just said “Waa!” instead of “Wow!”, and you wonder how long you’ve been doing so without noticing.
Gesturing is trickier; some of this sign language coincides with what you already know, sometimes it’s the complete opposite, and some gestures are so culture-specific that you couldn’t begin to guess their meaning.
Recognition of facial expressions involves some extremely complex neuroscience that renders another culture’s facial cues difficult to understand and almost impossible to spontaneously reproduce. Even illustrated faces- such as those of cartoon characters and emoticons in text messages– don’t always translate cross-culturally.