What is the size of the texture you are using.
I had a similar problem till I realised the textures used were larger than what I had set in preferences.
Go into your OFP Preferences from start menu. Then select Advanced - Textures.
I think it is the Objects one you might have to change.
e.g If you are using a 1024x1024 texture and your preferences are only set to 512x512 or 256x256 they look crap in game.
With .paa files, you need to use .paa files when using alpha channels.
What the actual differences are between them, I'm still not sure.
I use .paa for files with alpha channels and .pac for files without, dont know if this is the best solution though.
As far as tga and jpg textures go, I'm not sure.
I saw the thread on the official forums where Suma had some input.
I personally liked Colonel Klinks idea of using jpg for the first lod, then pac/paa for the rest, but I have had no experience here.
Hope some of this is helpfull!