Not necessarily. It depends on the lens.
With the 500mm F/4 the lens may be moving more mass inside.
Battery power, (other bodies vs the 1D line) seems to matter more for speed on the bigger lenses.
One stop is an 8 to 10 minute advantage in the morning and evening. It may not sound like much but you gain time at the most prime wildlife time.