The Kaaba is one qibla, but worshippers surrounding it approach that single qibla from different geographical directions.
Therefore a circular congregation cannot be assessed by distance alone as though all worshippers were standing in straight rows facing the same way. The key question is whether the follower is ahead on the imam’s own side.