If you want to keep the contact but you don’t want him/her to receive emails you can do that by changing the status of the contact to (Unsubscribe From All Lists).
So there is no need to delete the contact in that case.
Contact List Page
All you need to do is to click on Edit next to the contact you want, then click on Unsubscribe From All Lists:
Contact Activity Page
You are also able to do it inside of the Contact Activity Page. On the top-right dropdown select the Unsubscribe From All Lists option.