I think going with a 4x4 Xterra is a great route if you can find one in solid condition. I love Toyotas, but for the prices they go for used, you might as well buy a new one. I have a lot of experience with Xterras, and if you can find one that wasn't abused it's a great deal, very capable and you won't have to do much to them to go where you are planning. It will also have enought room for you and your wife if she ever decides she wants in on the outdoors. A used Subaru Forrester or Outback is also another great option if you are on an under 15k dollar budget. You can sleep out of it, they often come stock with crossbars so you just add whatever basket, bike, kayak rack system to the top without needing to spend 500-700 bucks on crossbars. They are also super reliable. I've owned two of them. I bought a new Forrester back in 2004. We racked 300k miles on it before the transmission went out, and only cost 800 bucks to get another one dropped in. Then I gave it to my niece as a present. She flew into the CA, and drove it back to Albuquerque back in 2016. She still owns it. Runs like a top, and takes her to all of her snowboarding trips in Northern NM and CO.
If you are certain it's just going to be you on these trips, I can't speak highly enough of the Ford Ranger. I owned a 96, and my buddy owned a 00, and those things are bomb proof from my experience, are very capable with slight modifications, and cost much less than a Tacoma. The best thing about the Ranger is that there are millions of them, and many of them live in salvage yards, so you can go to your local pick and pull and grab extra parts like shafts, CV's, body panels, leaf springs, you name it for dirt cheap. Same goes for any late model explorers. I will forever love these vehicles, I have been on my best trips in my ranger based vehicles and the cost of ownership per year was super low. Best of all, I had all the confidence in the world in them while I was out in the sticks with them.
Going used though, I will preface that you will either to know some very mechanically inclined people, or be handy yourself to give you the most benefit from these types of vehicles. The more you can learn to do on your own, the better. My all time favorite vehilce that I ever owned was my awesome 03 Sport Trac. This is, I believe the most underrated overland/camping/adventure vehicle out there. Mine was a beast, was cheap to modify and repair, easty to work on, and never failed on me in the field. And I took this thing everywhere. If you can find one that's an 03-05 in great condition with low miles, jump on it. All the benefits to the ranger based vehicles that I have mentioned above go for the Sport Trac too as it is also a Ranger based vehicle. Good luck on your search!
