مَعَ أُولِي الْأَمْرِ مِنَ الْمُسْلِمِينَ بَرِّهِمْ وَفَاجِرِهِمْ
With those holding authority among Muslims, whether righteous or sinful.
Barr refers to a righteous person; fajir refers to a sinful person. These terms describe the personal moral condition of the authority. That condition does not by itself extinguish the underlying prescribed status of Hajj or jihad.
Condoning someone's sin and recognizing that a separate religious ruling remains valid are different matters.