Some companies are now making small portable solar photovoltaic generators for just such a use. Here's an example: http://www.thermomax.com/pv_portable.htm
My guess is that your battery charger operates on AC power, but charges the batteries with DC (i.e., there is a AC to DC converter in the charger). PV is the opposite: it generates DC and usually has a DC to AC inverter to make AC power, which is required by most electric devices. Maybe someone makes a solar battery charger that cuts out the intermediate step of converting to AC. That would probably save on cost and efficiency.
Of course, it might be cheaper to buy as many batteries as you think you might need. PV is pretty expensive, and may not work well when wet.