Armsty's on the right lines. You need to put it other way round when you define something
prisonerGroup = group this
The activation field is right so the only remaining bit is the condition line of the trigger. You could have something like this:-
(leader prisonerGroup distance player < 5) and ("alive _x" count guardGrp1 == 0) and ("alive _x" count guardGrp2 == 0)
Syntax not guaranteed. Of course you may want slightly different conditions. If there is only one guard group it's easier - group the guard group to the trigger and you will get new options in the Activation box. Play with those - you might end up with
this and (leader prisonerGroup distance player < 5)
Now we come to the important bit of your question.
Is there a way to do this with a trigger or should it be scripted?
If you want to detect something happening during a mission, the best way is usually a trigger. That is what triggers are for.
Sometimes, the consequence of a trigger (the activision as LCD calls it) is controlled by a line of code, in which case it goes in the On activation field. If the code is too long to go there conveniently, it may be easier to get the trigger to call a script file and write all the consequences in there instead.
In other words, a lot of the time a script file is just a way of making the on activation box of a trigger a bit bigger and easier to use.
There are of course exceptions to that and there are things that can be done with scripts that can't be done with triggers and waypoints. But triggers and waypoints, perhaps with a couple of lines of code in them, should always be your starting point. Many official BIS missions have no scripts at all.