Lots of hosts get low location ratings for no good reason. Guests will give the stupidest reasons for a low location rating, like they wished it had been closer to the beach, as if the host can pick their house up and move it. I remember one host who had a stunning property with an infinity pool overlooking a pristine beach saying he'd get low location ratings from guests who were fussed that it was a 10 minute drive to the nearest store.
There are quite a few threads bemoaning this on the forum. But neither the location rating or the value rating are really anything to fret about. Airbnb will sometimes warn hosts if their cleanliness or communication rating drops too low, but location isn't any big deal.
You are being accurate about the mileage, but in many cases mileage doesn't tell anyone how long it would actually take to drive that distance. In my touristy little beach town, the traffic can be terrible and the streets are narrow- it can take me 10 minutes to drive 5 short blocks if I have to get across the tiny downtown core on a weekend. So if I were you, in addition to the mileage, I might also write the approximate time it takes, making it clear that would totally depend on exactly where in town someone is going and how the traffic might be.
But just because one guest acted ridiculous about it doesn't mean it will ever be an issue again.
BTW, if you ever leave a response you wish you hadn't, you can ask Airbnb to remove it- they will remove your own reviews or responses on request, they just won't remove a guest's review or response unless it violates their policies.
@John5097