Phonagosia is not quite the same as being unable to hear variety in people's voices. As the article states, people who have phonagosia can tell just as well as anyone else if someone is old or young, male or female, from their voice. What they cannot do is associate a voice with a person. Naturally this can cause difficulty in daily life, such as when speaking on the phone.
|Skye - telephone|
Of course, I also have voices for whom my hearing loss seems barely significant. I can understand my mom almost 100% of the time even in a loud situation. My dad, brother, and husband are also easier to hear. That comes just from talking to them constantly and from their knowledge of "how" to speak to me.