Kind of split between basketball and soccer. I've reffed basketball before, and it is VERY hard, even for someone who's played a lot. But a friend of mine is a college soccer ref, and it's extremely physically demanding, like others have mentioned, with more ground to cover, and almost as difficult as basketball in an of itself. (not to mention death threats, etc)
In basketball, you have the assistance of three refs, then the crappy video replay, then the crappy "light" above the rim, then the crappy "shot clock" which dictates how long the player should take to cross a line, then the changing of a rule every season (wait, now they can't put TWO hands on someone). So easy. SOCCER says hello.