I have taken a holding deposit for the private sale of my car. No written contract for the sale was written.
Being in writing is not critical to the existence (or not) of a contract.
That said, there may be rules in your state about dealing in motor vehicles (in particular) having to be in writing.
Upon further thought, I have decided not to sell the car and want to issue a full refund of the deposit the buyer has given me.
Vendors sometimes say this when somebody tells them something along the lines of
"...you could have got more for it if you had waited...", and so they want out of the deal.
(I do accept though, that people can have reasons for wanting to back out of a deal are indeed genuine)
But the buyer is now threatening me and forcing me against my will to sell my car with very threatening statements, refusing to pass me the banking details for the refund. Do I have my legal rights to withdraw my sales? I feel very threatened.
It's not against your will. You willingly made an agreement to sell the car.
You were so willing that you took a partial payment.
It's not like there was never a deal. You made one, and now you want to unmake it.
He's not "forcing" you to do anything. You've already agreed to do it.
What the buyer wants is for you to stick to the deal (in lawyer-speak, to "perform the contract").
Perhaps what you perceive as threatening is merely his frustration coming through
as he tries to persuade you to stick to a deal you willingly made.
A deal, fairly made, is basically a deal.