Often people tell me they would like to be healthy and slim but feel being heavier keeps them safe. What they are feeling "unsafe" about varies--- Jealousy from peers, relationship fears, abuse etc.
It is unfortunate that one might be subjected to some very unfriendly people. However the weight of a person does not predict or protect one from harm. I am not suggesting that we not take precautions, but one should do this regardless. I am suggesting that you should not deny yourself the physical body you want. If you are using your weight to "fly under the radar" , you are implying you are responsible for the actions of others.
As for relationships, there are plenty of people who are attracted to many different sizes. Regardless of what size you decide to be, someone will be a member of that club.
The only reason to dedicate yourself to becoming healthy and slim is because you want it. It's your life. Take responsibility for your own thoughts and actions. And let others do the same for theirs.