فَمَنْ كَانَ مِنْهُمْ أَقْرَبَ إِلَى الْكَعْبَةِ مِنَ الْإِمَامِ: جَازَتْ صَلَاتُهُ إِذَا لَمْ يَكُنْ فِي جَانِبِ الْإِمَامِ.
Translation: A follower closer to the Kaaba than the imam may validly follow, provided he is not on the imam’s side.
The decisive qualification is “provided he is not on the imam’s side.”
Distance alone does not decide the issue; the follower’s side relative to the imam matters.