That was a thought I had, as well. But empirically...
- RF 800mm, f/5.6 is 461 mm long, f/11 is 362 mm long
- RF 600mm, f/4 is is 472 mm long, f/11 is 270 mm long
- RF 100-300mm f/2.8 is is 323 mm long, RF 70-300mm f/4-5.6 is 223 mm long
So it seems that in practice, aperture matters quite a bit. You could certainly argue that IQ matters as well, but I have no doubt the IQ of a PowerShot V lens would be more in line with that of the latter lens in each pair above rather than a big white L lens.