In the same expression, the speaker may intend the overt addressee and also warn another person. The literal addressee is then not excluded: literal reference and an additional indirect aim coexist, making a kināyah analysis possible.
The contrast is:
Only the other person is meant — a figurative shift in address may arise.
Both are meant — the original addressee remains, together with an indirect target.
Context, not the bare grammatical form, determines which reading is justified.