The right time to refuse a search is always
When should you refuse a search? Always. To do otherwise is to erode privacy rights, one concession at a time.
A friend posted a video of a girl being tased because she refused a search and lipped off to a border agent (this was in the US, the friend is American, the commenters were a mix of Yanks and Canucks).
Predictably, one comment was to the usual effect, that if she had nothing to hide, she should not have refused.
Well, that got me riled. After I took a deep breath, this was my reply.
The only reason to refuse a search is to protect your privacy. The question of whether or not you are hiding something is irrelevant.
There is a reason you (Americans) have a Fourth Amendment and that we (Americans, Canadians, many others) all have prohibitions against unreasonable search and seizure.
The starting point is that any and all searches are unreasonable, invasions of privacy. It is then up to legislators and the courts to specifically and reasonably allow searches under specific and reasonable conditions, balancing the privacy right against judicial oversight and probable cause.
Refuse all searches, because you are American. I will refuse all searches because I am Canadian. This is what it means to defend freedom and liberty: Not with a gun, but with a stand. On principle.