well, I got the 03 double cab 4x4 with the 3.4L V6 Engine. With a loaded Bobcat, wife, two kids and dog we might be a little heavy and overloaded, but it does work.
The V6 is nice and I usually don't go faster than 65 mph, usually less than that. Mileage without camper is about 18 to 20 m/g (driving really carefully). With camper this goes down to 16 and less. On our last trip through southern Mexico and the mountains I only got 11 m/g with an average speed of 24 miles p hour!
I think the 4 cyl should be enough for most of the year without the camper, for the few times you take the camper out you just relax and take your time.
I've seen Hilux and Tacomas in central america and the Caribbean working for years, way overloaded every single day on really bad roads. They crawl and smoke up the hills but they get where they have to go.
The poor mileage of the 6 cylinder engine is an argument in favor of the 4 cylinder. Speed and acceleration are not that much fun when you have to pay at the pump. But, in 03 the 6 cyl was the only engine for the double cab with 4x4.
So, for me I would choose the 4 cylinder, it's just the little engine that could.