Step 1 is to have a frank and friendly discussion with your neighbors, invite them out to coffee or a cocktail and discuss it. You need to get to the bottom of what their concern is so you can take appropriate action.
You don't provide enough detail if this is a co-op, rental or condo, so there may or not be some protections in your by-laws or lease against against any harassing behavior by a neighbor to you or your guests (I am assuming you have the right to ABB your unit).
It also could turn out to be nothing and you may find out that they have unfounded issues, like security and there may be ways that you can alleviate their concerns. Maybe them just having your phone number so they can call when something is actually going on, may make them feel more comfortable, especially if you show you will take quick steps because you don't want that to happen either. Perhaps, it is about the noise and putting down carpet or area rugs may quiet it down or taking out the loud sub-woofer. Showing that you respect their feelings and care will go a long way.
It may not sound palatable or maybe even like extortion, but maybe you pay them for the perceived inconvenience of allowing ABB guests (maybe 1%, makes them deal with the noise) .
Don't ignore it and don't escalate it, de-escalate it, everyone may end up with what they want.