If you really, really want to make sure you aren’t pulling in any parts of X when installing dports, and you’re building from source, there’s a few options you can set to keep X11 off your system.  You can even go farther.