A transfer to figurative meaning does not mean that every remotely associated thing may be inserted into the oath. The intended substitute must have a recognized relation to the word and to the action described.
Tree: the edible fruit is the normal candidate.
Pot: the food prepared in or taken from it is what the statement concerns.
If literal consumption is realistically possible in a different context, one must reconsider the claim of impossibility and the role of convention. The classification depends on the actual statement and usage, not on a desire for the most convenient interpretation.