Drafting/tailgating is always something I'd fought companies on. If I'm responsible for physical security, arm me to protect myself when I go and challenge someone and things go sideways. Usually the conversation ends there, but one time I actually did threaten to take it a step further when someone tailgated me, and I told them to stop, or I will call the cops for them trespassing. That someone was a board member without a badge to the building. Good thing tenure and security policies override a board member trying to get me fired. They stopped pushing the aggressive anti-tailgating policy after that.
With security teams, I always find that they are gung-ho about implementing systems and policies, but evaporate when challenged to enforce them or actually do the real work related to those policies (like monitoring cameras, staffing for physical access security, etc.).
I once automated a co-worker out of a job with a 13-line bash script. I felt guilty on a human level, but they were also awful so that guilt faded pretty fast.