Jealous of old flame?

I have been dating this older man for about 2.5 years and we get along extremely well most of the time. Every once in a while we have an argument, but very seldom. Anway, his ex-wife calls him on occasion to remind him of his daughters (not their's together) birthdays or if someone that they know has died or something like that. Anyway, it used to bother me when that she did that and I said something to him about it. He told me that there is absolutely nothing going on and that they get along better as friends now than when they were married. And he told me that I wasn't going to tell him who that his friends were. After much thought I realized that what he was saying was right. He owns a restaurant and I work in there for him and now he gets jealous of some of the men that come in there. My first love comes in there a lot and yesterday he was making little comments about him. I just shrug it off, but it appears that he is now somewhat jealous. I told him that I cannot control who comes in there and that he has said nothing at all out of the way to me. Is he afraid that he is going to lose me or what?