Well, first off, just as everyone has the right to decry whatever they will, I can decry the decrying itself, and I don't think it automatically becomes incumbent on me to delineate exactly where enough is enough--anymore than it is on those who are outraged to introduce some nuance themselves.
But if I had to draw a line, it would be to let him fail in the natural way a comedian fails: empty seats. That's what I meant by vote with your wallet. If you don't like it, don't go to his shows, and if enough people don't like it, then maybe his "comedy" was geared towards something besides just being funny.
As it stands, he had a large enough audience to earn that role of Oscar's host. But people that were offended by jokes from a decade prior came to pressure a third party to drop him. This is less voting with your wallet than it is direct protest. He's not failing on his merits since he's successfully doing exactly what his job entails, making people laugh and selling out arenas.
The same thing with Shane Gillis. He grew his following and career until the point that he was hired to SNL, and then they dropped him because of protest. After the initial outrage died down, he was later brought back to host, and everyone seemed to be okay with it.
In every case, these performers were facing real-world consequences for what they said in their capacity of comedian--it wasn't liked leaked audio or something--and they faced career setbacks despite doing their job well.