وَاسْأَلِ الْقَرْيَةَ الَّتِي كُنَّا فِيهَا.
Ask the town in which we were staying. (Yūsuf 12:82)
One does not literally question the town's buildings; the relevant people are its inhabitants. Reconstructing “the people of the town” is a well-known grammatical explanation. The rational and customary clues supplied by the verb “ask” guide the reader toward the appropriate referent.
Not every verb paired with an inanimate noun automatically proves an omitted genitive head: figurative attribution and other constructions are also possible. In this particular expression, identifying the townspeople explains how the requested testimony can be obtained.