It just doesn't need to be so precise. The more the timing is kept informal, the less stress for everyone.
For example, if you start timing things down to the millisecond, you'd also have to do equipment scrutinising, like they do in motorsport, to make sure everything is at the same level. In Formula 1 there have been drivers disqualified from races due to tire pressure not being in the right range, stuff like that.
Some similar in speed running could be questions raised about power supplies to consoles. If an NTSC console was fed power at 60.5hz instead of 60hz, that tiny difference could add up over the course of a long game and soon be significant.