In wa-lākin nafā ʿannī al-rijāla jarāʾatī, nafā is the past-tense verb, al-rijāla its direct object, and jarāʾatī a delayed grammatical subject. It does not mean the men drove away my courage; it means my courage drove the men away from me.
Iqdāmī, ṣidqī, and maḥtidī add further qualities to that causal list. Recognizing subject and object order determines the correct meaning of the verse.