Back in the NES/SNES days, I used to wash my hands before I fought the final boss of any game. I had a tendency of gripping the controller really tightly and would sometimes get sweaty hands, so I would wash them and wipe down the controller before fighting the boss.
The real problem these days are the seams of the controllers where gross shit accumulates. I often use pointy sticks or pieces of paper to scrape it all out.
Yeah, I hate it when shmutz accumulates in the seams of controllers. I tend to either use a toothpick to clean them out or get one of those disinfectant wipes and run it all over the controller. Also, a can of compressed air helps.