I agree with you that the ACLU should not be considered a special interest, but suspect that many of my Republican friends might disagree.
I think the ACLU certainly walks a fine line. They do a great job fighting for rights even when they are unpopular, but I think it's hard not to argue that many members of the ACLU slant towards the left. They also have become a bit more active in rights than defensive of those that exist. At least it seems that way to me.
A lot of ACLU staff slant libertarian too.