It's a bit random, but I found that if I use these, with a bit of rubber tape around them to take up a bit of room, I at least have a solid core to tighten down on. I only use the cylindrical part... I removed the extra junk. You can get this item at home depot in the fence section.
I've actually used these and with the set screws, AND drilled holes with lynch pins to keep everything attached. It's not as tight a fit as you can get with hardwood dowels, but dowels require more work to fit perfectly and they require oiling to stay the same size.