Being on the imam’s side does not mean standing at the same exact point; it means sharing the side of the Kaaba toward which the imam faces.
When imam and follower face the Kaaba from the same side, precedence can be measured in that common direction. Where they face from different sides, the ordinary straight-row comparison does not apply unchanged.