If you are on DragonFly, using pf, using altq, and using fairq to control usage, there’s a latency bug that Matthew Dillon recently fixed.  He’s posted an announcement and committed fixes to master and 3.8, so it’s only an upgrade away.