I don't think it's rude. It's a favor, after all, not expected behavior.
I almost always respond with, "depends on the favor."
They could be asking for you to take their picture; they could be asking for $1000 for their MLM. I'm not signing a blank check by answering "yes".
I think it speaks to his style of arguing. He doesn't commit to anything, so he can't be caught in the wrong on anything. He's free to take pot shots at his opponents while never giving them an opportunity to hold him accountable for his wrongheaded BS.
One of my college friend's FAVORITE musician was the Boss, and she refused to go to his free concert opening for Obama. It was hilarious; she was absolutely torn in half all weekend.
Sorry, but I prefer this to winter days where I don't see the sun at all because it's only up while I'm at work.