Oh man! That has got to be super annoying! Are you by any chance using a Gear Watch with your Note 5?
Can you please try these steps on your phone?
1. From the home screen, tap on Apps.
2. Tap on Settings.
3. Tap on Applications.
4. Tap on Application Manager.
5. Tap on Menu.
6. Tap on Show system apps.
7. Search for S Voice application
8. Tap on Storage.
9. Tap on clear cache and clear data.