Drifters rarely can ever beat you in an all out road race. Once you've trounced them, they cry about how your such a boring and untalented driver because you can't drive through an entire coursewith only two wheel of traction.
I'd say yes. Might be a bit more stabilizing (braking understeerish) if you are going for REALLY good handling, and it may make trail braking a bit stubborn on turn in, but definitely worth every bit for the rest of your driving experience in the real world.
ABS is always left on when the Corvettes run at the Nurburgring. If it were a really big detriment to handling, you'd see them disabled when running true track evaluations.