The sermon addresses the people; during du‘a the imam turns toward the qiblah. These are distinct actions.
The report of ‘Abd Allah ibn Zayd includes وَاسْتَقْبَلَ الْقِبْلَةَ
Translation: And he faced the qiblah.
This supports facing the qiblah when supplicating.