You can find more than a few caves when you hike up to the Redwall and Muav limestone layers. They are pretty much the same Redwall caves as you'll see from the river when you're in the mile 25-40 area, going by the Cave Springs caves at river level, Stanton's Cave, and the Royal Alcoves and such. The tricky part is that when you reach the Redwall many times you'll have a cliff in front of you, not a slope. But, not every time. Go explore!
Specifically, just above South Canyon, on the left, is a small pull-in at a slope up to where you can find what you are looking for.
Also, Tapeats Cave is up Tapeats Creek, and you'll find more if you walk up to the right from the creek and also left, up and over into Surprise Valley.
One of the good things I like about heading out of camp and taking an unknown, to me, walk is finding something unexpected. Many interesting things are waiting to be found. "Hey guys, let's take a walk here and see what we might see." Ditch the guidebook hikes; go write your own.
A related walking idea is to go to the Coconino layer the first day or two of your trip and look for fossil footprints in the broken chunks. They are there!