Figurative uses are not all based on the same relation. When an expression is transferred because of resemblance, the figure is metaphor (istiʿārah). Where the relation is causal, part–whole, container–contained, or another non-resemblance relation, the figure is a non-metaphorical transferred use (majāz mursal).
رأيت أسداً يخطب
I saw a lion preaching.
Courage links the orator and lion. By contrast, naming a location to refer to its inhabitants depends on a spatial relation rather than likeness. What matters is the type of semantic relation, not nonliteralness alone.