Inside the Ka'bah, worshippers may face different internal directions because they are within the Qiblah structure itself.
In congregation:
• if the follower's back is toward a direction other than the prayer leader's face, the following can be valid;
• if the follower's back is directly toward the prayer leader's face, the following is not valid because the follower is effectively ahead in that direction.
The issue is relative position, not a claim that one wall is sacred and another is not.