The only two things I can think of is one the guards may be moving towards a guarded by west where they go to the trigger area defined guarded by west. Two the guards waypoints might need tweaking. What's sentuary?
The hold waypoint should be solid but maybe its your other waypoint? Try diff behaviors too if the guards are not going to be in combat, just eye candy like CARELESS should make them think the enemy is their best friend up unitl being shot.
Actually one more thing might be the problem. Depending on what island is being used, some islands have buildings that oriented to a side. Like some castles will come up as valid targets that are red cause they are castle east etc.
Your guards may be running from that. To detect what's going on, be a squaddie in the guards group (your a follower) The guard should report what's wrong, (Contact! Enemy castle at 12 o clock!)
.