That's a double-barrelled question.
On the one hand, a parent can withdraw their consent for a child to be taken out of Australia.
Especially if they have bona fide concerns that the child might not be returned*,
or if there are particular (and again, bona fide) concerns for their welfare.**
Taking possession of the children's passports might (very, very "might")
be a reasonable move in those sorts of circumstances.
On the other hand, taking possession of a person's passport
with no other intent than to cause them (or someone associated with them)
financial loss (such as losing the money they (or a parent) have spent on a pre-booked holiday),
might amount to stealing (of the passport), which could indeed be a police matter.
In any event, stealing or not, without your consent, I cannot imagine any basis at law
for him to to take possession of your passport. That could well be a police matter.
Hopefully some of the lawyers here (most of whom know far more about Family Law than I do)
will have some more helpful suggestions.
-----------------------------------------------------------------------------------
* For example, if doing so would have the effect of confounding parenting arrangements,
or obstructing the work of the Family Court.
** For example if the child was being taken overseas for an arranged marriage,
or to a country where they are exposed to cultural practices that will cause them harm,
such as FGM in some parts of Africa.