I don't do Family Law, but it seems to me that, absent any family law factors,
you (the prisoner's mother) don't have any "rights" in this at all.
It's a well established premise of Family Law that, where possible,
a child has a right to a reasonable relationship with both parents.
That is a right of the child.
Not the parent. And not the grandparent.
If their mother is their principal carer, then where they go (or don't go)
and what they do (or don't do) is pretty much her call.
Their mother may believe, quite reasonably, that the experience of attending the prison
may be distressing, or even traumatic, for them.
It may be that the children don't actually want to go, but their mother is
unable, or unwilling, to say so to your face.
Their mother may, quite reasonably, be unwilling to expose her children
to (other) crims, and/or, dare I say, to the families of other crims.
I am not at all sure that you, as the grandparent, are even on the field of play here.