The trigger that ends the mission should have the Condition field:
SquadSize - {alive _x} count units group player >= 3
Initialise SquadSize at the start of the mission.
An alternative is to give each of your squad a killed Event handler that increments a counter.
In init.sqs put the line:
Casualties = 0
in the player's init field put:
{_x AddEventHandler ["killed",{Casualties = Casualties + 1}]} forEach units group this
Make a trigger that ends the mission with the condition field:
Casualties >=3