The land is expensive.
This is how they look at it ('they' being prospective track builders/owners/businesmen):
1.) I have $10mil to spend that would bring in $50k a year at a track.
2.) I have $10mil to spend that would bring in $1mil a year in rent/mortgage/etc.
Choice B wins every time.
If you're only bringing in 50K/year running a track, you're doing it wrong. Your point is correct, though.
It takes people who live, and breathe motorsports to build and develop a new track. There's just not much money to be made in the business anymore, it seems. Turn on the TV on Sunday and you'll be hard pressed to find a race with the stands full. Maybe F1 is full. Maybe NHRA is full because they only hold a few thousand fans. But that's about it.