Children's matters are very delicate - parents can sometimes be seen as supportive by facilitating time with a father they don't trust; sometimes they can be seen as overbearing or controlling. It's impossible to predict.
Realistically, you're dealing not just with distance, you're also dealing with family violence, which in many cases leads a judge to order supervised time with the violent parent, rather than unsupervised time. There is no real way to monitor whether that time would be supervised, though, given the fact that you live in different states.
So with all that considered, my personal, very non-legal-advice opinion is that you should tell the father you are willing to facilitate the kids spending some time with him, but not before both of you attend mediation and get a parenting plan on paper. At the moment, while there's no agreements in place, either parent can do what they want, including keeping the children, but with an agreement on paper, you will have something to show to a court if you need to get an urgent recovery order.
So in summary, tell the father you're willing to facilitate time, once you reach an agreement about parenting arrangements through mediation. Contact Relationships Australia and organise a family dispute resolution conference - it will be up to them to contact the father. If he doesn't respond, you get the certificate you need to show you've made a reasonable effort to reach an agreement, to no avail. If things continue, then you might consider pursuing orders through court.