Thwarts help keep the boat from folding up in whitewater and shouldn't make any difference in the height from the floor and the top of the tubes. If you have a frame on the boat, you don't need them because the frame provided the stiffness to the boat. Unless you've got a good reason to haul a few cubic feet of air from the put in to the take out, take your thwarts out, remember where you put them, and only use them only when running as a paddle boat.
Since 2004 I've only used the thwarts a couple of times in my boat and only then when using the boat as a paddle boat.
Nothing in the world is more yielding and gentle than water. Yet it has no equal for conquering the resistant and tough. The flexible can overcome the unbending; the soft can overcome the hard. - Lao Tse