In Vic the model rules for OCs are silent on this issue. Check to see if you operate under the model rules or some variation of them.
If your rules are also silent, you should not attend unless invited. You can be deemed a trespasser if the committee meeting is held on private property and you have no invitation, especially so if you have been told not to attend.
QLD is weird, they have different rules depending on the type of premises. While non-committee members under the 'standard module' can attend, there are limitations around their attendance including giving written notice at 24 hrs before the meeting, and keeping quiet unless addressed by the committee.