Regardless of how much land you have to use.. if neighbors complain about noise and neighborhood regulations don't like your use of the land, you can have problems. Better to keep it a dirt track and have fun with off-road gocarts/bikes/paintball/etc. I built one a few years ago with some used parts from broken ATVs, plus a 1300cc Suzuki motorcycle engine/transmission from a wrecked bike. It's badass to have a mini dune buggy with a full steel rollcage roostertail 0-60mph in less than 6 seconds on dirt.
Paving a track costs a TON of money - just look at how much it costs to pave a driveway. Getting rid of it later is even more expensive. Keep the land natural and have fun with it.