After a completed sale, the seller still owns the tree, while the identified fruit belongs to the buyer.
Buyer's right: to take possession of that fruit and remove it lawfully.
Seller's right: not to have someone else's property occupy his tree for an indefinite period without entitlement.
The removal rule protects the boundary between two separate ownership interests; it is not simply a penalty for purchasing unripe produce.