To strongly withdraw one's hand from someone, or not let someone hold one's hand.
To reject someone's suggestion strongly.