The differences is that the refs very rarely get involved on a play and even then it's unintentional and they try their best to stay out of it. It's not like it's a static object in the field of play. The players also aren't supposed to intentionally get the refs in the play either. It's not like they are allowed to use one as a pick if they feel like it for example. The refs being out there are more a necessity for the game to function than some unneeded luxury that might cause problems.
Unlike baseball where every field's dimensions are different and it's a natural part of the game, I think football would loose something if it didn't keep things more or less standardized to keep things equal.
The refs are considered part of the field....just like an unusually slick spot or something. The point is if it's there an you know it's there it's your job to avoid it. You don't get a do-over if you hit the ref with the ball, slip on a slick spot or punt the ball into some jumbo-tron IMO. It's not unreasonably obstructive so I think it's just (or should be) an intrinsic part of playing in that stadium.