I would imagine that there is a vast difference in the level of skill needed to play Quidditch and that needed to be a referee. You wouldn't need fancy skills to referee, just an ability to get up and down the pitch with play and possibly a few sharp turns. I think he must be able to fly well enough to do that or else I'm fairly sure that Harry and co would have been snickering at his efforts.