Indeed, "House rules" are in any practical sense, unenforceable, so in my experience, you need to resort to a bit of simple psychology.
First, it's important to have some degree of contact with the guest and establish a friendly relationship with them. If they never meet you, then you're faceless. They don't know you and are unlikely to consider your interests when making or leaving messes or ignoring house rules.
Second, most (decent) guests don't actually realise that house rules are unenforceable, so attaching some sort of penalty to violations of them can make them think twice. "Smoking in the house will result in a 500€ fine". Yeah, well, you may not be able to enforce that, but they don't know that. It makes them think twice.
Third, and I can't say this often enough... Screen your guests. Ask questions before they book. How many people? What is the age range? If they seem unusually detached, don't answer, avoid answering, or give inconsistent answers, that's a red flag, and you have to decide for yourself whether you're willing to accept this guest. I can tell you from long experience that it's spared me regret quite often.
And then, there's a security deposit, if you feel uncomfortable with the guest who's just made a booking request, you find out it's a bunch of 20 yr olds, and you'd prefer to either scare them away or demand a big security deposit... It either scares them away or gives you security. If you're holding a thousand bucks, you can be sure they'll be extra careful. Often, they'll go to great lengths to leave everything in perfect condition. More than you ever expected. They want their deposit back.
It's all psychology, really. And yes, it's work. But anyone who tells you that hosting is easy money has obviously never been in the business...
Which reminds me of those Airbnb hosting adverts I hear on the radio. "You can put that extra property to work and make extra money. And you're top to bottom covered by Air Cover", so it's all easy peasy.
Yeah, right then. So, Airbnb, you've never been in this business eh? Allow me to give you a few pointers...