The delegation described its own local situation and worship site. The Prophet’s ﷺ instruction was directed to that particular group.
Principle of interpretation: A historical directive needs its addressees, ownership, circumstances, and purpose. A specific incident cannot be turned into a general authorization to interfere with others' religious sites or property.
One clear lesson is the delegation’s practice of asking for guidance and seeking clarification about practical obstacles before acting.