This would completely creep me out, too.
Did he break into your room or do your guests have access to your private living spaces?
I am certainly not blaming you (the victim), but we all need to take safety precautions as hosts --- safety for ourselves and our belongings. Women especially have to think of the worst case scenario and work from there to help prevent it.
Thank God that I haven't had any creeps that would make a copy of a key. Two weeks ago, I had a locksmith install a second lock on my front door. Guests get a key to that, but not to the other deadbolt which I lock when I am not here. My private living area is locked 24/7, too.
Please change your locks ASAP and lock your bedroom when you are home and when you are not home.
The sad thing is that this will probably make you wary of single male guests which is a shame. Not all men are creeps.