I could share my experience, but it won't be nice.
I have bought many houses.
Neighbors ALWAYS tell me what I can and cannot do on MY
property.
"You can't build up" (I could)
"You can't park on the street in front of my house" ( I could)
YOU CAN'T CHANGE MY NEIGHBORHOOD, BECAUSE I DON'T WANT YOU TO!!!
Neighbors are greedy, just because they think they can be. To them, it's free.
Fortunately, we live with a Constitution that allows PRIVATE ownership and due process.
As for your neighbors, I don't have a lot of advice. Bring them baked goods? Ask about their children?
Feel them out.
Do they have LEGITIMATE concerns? or are they just lonely?
Smile and be firm in your ownership rights.
In general, beaurocrats hate dealing with these neighbors' complaints just as much as you do.
Good luck