Choose gender, number, and case together:
رِجَالٌ أُولُو عِلْمٍ — masculine plural, nominative.
مَرَرْتُ بِرِجَالٍ أُولِي عِلْمٍ — masculine plural, genitive.
نِسَاءٌ أُولَاتُ عِلْمٍ — feminine plural, nominative.
رَأَيْتُ نِسَاءً أُولَاتِ عِلْمٍ — feminine plural, accusative.
The mudaf-ilayh عِلْمٍ remains genitive in every example.