Hmm, the icon won't denote the number of messages, but there should be a little icon in the top nav that shows there is a voicemail. In your situation, is the icon missing entirely?
Did you need more help? Please let us know if you've taken a look at Lauren's reply. Thank you.