ذُو means “possessor of / having” and is always mudaf in this lesson. It attaches only to an explicit noun:
رَجُلٌ ذُو عِلْمٍ — a man of knowledge.
رَأَيْتُ رَجُلًا ذَا عِلْمٍ — I saw a man of knowledge.
مَرَرْتُ بِرَجُلٍ ذِي عِلْمٍ — I passed by a man of knowledge.
Do not attach ذُو directly to a pronoun in this taught construction.