The Bat Jar Conjecture - S1-E13
Sheldon: At this point, I should inform you that I intend to form my own team and destroy the very bonds that bind your very matter together and reduce the resulting particulate chaos to tears!
Leonard: Thanks for the heads up.
Sheldon: You're welcome. [Turns to leave.] One more thing,
Leonard: Yes?
Sheldon: It's on, bitch.
Answer: Because Raj's selective mutism seems to be caused by women that Raj sees (on some sort of conscious or subconscious level) as a potential sexual partner. An attribute that wouldn't apply to either his sister or mother.