If both imam and follower stand on the same side of the Kaaba and the follower is farther forward toward it, following is invalid: he is visibly ahead of the imam.
On different sides, mere relative distance from the Kaaba does not establish that legal precedence.