Ballistae are relatively simple as far as range is concerned - if you can click on and target your enemy, you can fire at your enemy. The tricky part with Ballistae is using them to fire at higher or lower elevations. I used one to take down a Trebuchet up on the ramparts within Stonemist Castle a couple of nights back, but found that I could only hit it with the No.2 AoE bleed shot - the other two shots just impacted with the platform it was sat on. Likewise for defence, you will find that if you don't set them on the very forward edge of a wall (where every enemy AoE ability can target them), their shots will simply be blocked by the wall for anything but the furthest ranged enemies (occasionally useful for enemy siege set up at a distance, but not much else).
In short, Balistae tend to prefer targets on a level field whenever possible.
Trebuchets are a case of trial and error. Often you will need a second player to be a spotter and help guide your shots to their target when firing at extreme range. If no spotter is available, just keep adjusting your shot until you start seeing little white damage numbers popping up. The site Silas Gray links is a great resource for figuring out where to place the Trebs in the first place, though only as an initial guide as the range can vary a bit - the higher your elevation to your target, the further your range.
Hope this helps.