I guess the majority of the people who will post here will tell you the guy was an ass and that you did the right thing. I'm also guessing that the majority of people who read this and don't post a reply will think you overreacted and that people are now talking about you behind your back at work. The guy was wrong, but probably thought he was being funny. If you guys are on equal footing socially, he will probably come out looking better in this situation. Not saying he was right........just that (at least the way you described it) you came across as being........well........that person at work.
On an unrelated tangent, have you ever gone to someone's house who has a teenage boy living there and then ask that boy if you can take a look under his mattress? The look is priceless.