If a man possesses only a pure silk garment, he is strictly forbidden from praying in it and must pray naked instead.

Exposure of the Awrah and Impurity on Clothing