From somebody who has never owned or wheeled, and certainly has never traveled, in a full-size, I'll go ahead and say yes, if you're hardcore wheeling, the size of your rig is going to limit where you can take it. That doesn't mean you won't find plenty of places to go, on and off the road.
The truth is that nobody hardcore wheels a full-size rig because of the size being the limiting factor.
If you want dirt roads and sand and mud, have fun with whatever suits your needs - but if you're going to run forests and rock gardens and high-clearance, tight trails, you will need a small rig. Modifying a full-size rig for decent clearance makes it huge, and bigger is not better in most off-road scenarios.
For reference, one of my pals has a 2020 Ram 1500 and I have a 1999 Ford Explorer. Turning around in the space of 1 lane's width is some thing that he has to live without, and his breakover angle is a laughing matter (he's at stock height with running boards).
On the flip side, taking more than 4 people anywhere is not an option for me, and forget bringing gear for even a weekend out - my maximum payload is around 900 pounds.
So if all you're doing is traveling, camping, and finding good places to get out of the car for fishing, hiking, kayaking, or the like, and the hardest terrain you take are dirt roads and sandy beaches and forest service roads, a full-size is for you! If you find that it occasionally doesn't clear obstacles, lifting is always a future option.
On the other hand, if you find yourself high-centering or tagging the rear bumper on departure, or struggling to find places to turn around, in your current rig, then consider that it will only be harder in a larger vehicle.
There are many, many people who prefer a full-size rig because their capacities and specs are much, much higher than a "compact" truck (which really aren't that much smaller these days).
Let's just say, most people don't regret going full-size, if travel is primary use and wheeling is secondary use.
