take a look at the attached screenshot.
maybe you made these smooth edges when you deleted some parts with an eraser tool with smooth edges like on the screen. at these edges the alpha value is
between 0 and 1 thus semitransparency.
what you did now was converting them with paatool and check rgba5:5:5:1 so the format would only accept alpha values of 1 or 0.
generating sharp edged alpha areas from masks works better btw