لِأَنَّ «إِلَّا» لِلْإِخْرَاجِ، وَالْإِخْرَاجُ يَقْتَضِي خَرَجَ مِنْهُ عَامًّا
Illā is used for exclusion, and exclusion presupposes a general domain from which something is excluded.
If no range of alternatives is understood, calling an item an exception would do no useful semantic work. The implied range must be broad enough to include both the retained item and alternatives to it.
Negation first denies the members of that range. The expression after illā is then exempted from denial, and restriction follows.