Many owners of the awesome little Nokia E62, have one of the same complaints I do. The stupid rubber audio jack cover is not connected to the phone, thus prone to loss.
To keep it with the phone when you remove it, try this. Store it between the battery and the SIM card under the battery cover. The space between the two is enough for the rubber lip of the plug, and the battery cover fits right over it without any issues.
Nothing ground-breaking, but simple and effective.