One night, a husband and wife were watching television together, when a fictional couple on the show "Friends" agreed to create a "list". The "list" would contain 5 people that each person was allowed to have sex with, without consequences, if the situation ever arose.
The wife watching the show with her husband thought it was a hysterical idea, and decided they should make their own lists. "It's just for fun. A fantasy. You can make a list of any 5 people, and if you ever get a chance to sleep with them, I won't be mad. Seriously."
The husband, hiding his sadness, reluctantly agreed. The wife (almost too quickly) filled out her list, still laughing at the idea, and handed it over for her husband's review. Jude Law. George Clooney. David Beckham. Jake Gyllenhaal. Ryan Reynolds.
The husband quietly read his wife's fantasies, then handed over his own list, only two entries filled out. Your sister. My secretary.
The moral of the story: Don't play games like this. Don't talk about celebrities being hot in front of your spouse. Don't rationalize it by saying "it's not like I'd actually do anything about it." It's not only the action that's wrong. The long-term goal is for that desire to not even be there.
And even if comments like that don't bother you, odds are, they probably bother your spouse. Even if only a little, why do it? And honestly, if they don't bother you, they should. You shouldn't take pride in the fact that "you're not a jealous type" or that "you trust your spouse."
You should absolutely be jealous of your love.