Property settlements are governed by the Family Law Act 1975, which is federal legislation over which the Family Court of Australia and the Federal Circuit Court of Australia have jurisdiction. State courts don’t have jurisdiction over family law matters since they’re not governed by state legislation (with the exception being some circumstances associated with amending parenting orders, which is not relevant here).
The points about your case that are important are that if the debt exists, it would be considered a shared debt from the marriage. This is because it was for renovations to a property that was considered a part of the shared asset pool between you and your ex. As such, if that debt exists, then it should have been included in the shared asset pool that was divided during your property settlement. It wasn’t and that property settlement has since been finalised. Your former brother-in-law can’t now come to you to recover a debt that belonged to both you and your ex-husband, regardless of the fact that your ex is incarcerated and regardless of whether your ex acknowledges the debt exists. You would have both been liable had it been included in the property settlement, and that doesn’t change just because your ex is in jail.
I wouldn’t reopen the matter in FamCA/FCCA. It’s your former brother-in-law raising the dispute, not you.