I don't take mine off either. I went a while without wearing one because the original slipped off and was lost. But, my father gave me his. He never wore his because he worked as a contractor and it's not a great idea to wear jewelry when doing manual labor.
I take mine off far too often. I take it off when I play sports, when I'm washing dishes, when I'm showering, when I'm sleeping. Most of the time I take it off for fearing of losing it while doing some of these activities. Has that stopped me from losing it on two different occasions in about 21 months of marriage? Nope.
I wear it to work and usually end up taking it off. I wear it to dinner or if we go somewhere, otherwise it's in the box. I have the wife's name tattooed on my ring finger and does she, no reason to wear it all of the time as I hate jewelry.