I really fancied this guy (double my age), years ago when I was single. I always got this strong feeling he fancied me or we shared some chemistry from the way he behaved around me, despite he shouldn't, (being married & in a very professional & responsible position), that he could've lost his job if he got involved with me.

Yes, it's not a very wise situation, it would have to be a very discrete affair. 

At first I didn't like him  because he annoyed me & we used to argue. When I tried to annoy him back, it surprised me he liked it & the flirting started from there!

Some men do like a woman that argues back, looks like you found a man that does. 

When around each other we couldn't stop smiling, take our eyes off each other, or when passing we often gravite towards each other to talk & I also often made him laugh & blush.

Over recent years he's noticed me a couple of times alone in town & tried to get my attention but spoken. One occasion he came up to me & we stared each other in the eyes for ages. When we first looked at each other, he raised his eyebrows at me, but did not smile & I noticed his eyes dilated the moment our eyes first locked.

Raised eye, I like what I see so I'm going to open my eyes wider to get a fuller picture and then the dilated eye just confirms that.

I got really nervous & couldn't speak, also I'm not a confident person & afraid I got the message wrong -giving me a come-on or another reason for approaching me, so I let him walk away (even though he still kept staring intensley at me).

Also I am married now,

Even more reason to avoid a relationship. 

but when I saw him again after years, the old feelings came back, wanting him, but confused what to do next. Later I decided to send him a note at his work, apologising for not saying hello as it really upset me we hadn't spoken, making the excuse for not speaking to him, that I hadn't recognised him. But it was to cowardly cover up embarrasment I was so nervous & excited when I saw him, frustratingly I got tongue-tied.

But now when we do see each other, he seems he's not interested in talking & looks away.

He may have thought you were not interested or equally he might have come to his senses and realised it's not wise having an affair with a woman when he's married and in a good position at work. 

I feel hurt & confused what has happened.

You shouldn't be hurt and confused, you are married and should be enjoying your marriage.  You could go on forever thinking there are better options around you but  then what usually ends up happening is you will split up with your husband and end up having a series of very short lived relationships and end up on your own.  I know the modern day view is to seek excitement and a new partner every so often, but the truth is it isn't really as glamarous as it looks on the TV.  Most of the time you end up having a brilliant time for a few months, followed by a seriously distasterous time.  If you do leave your marriage it has to be because you are fairly certain about a guy and you've spent a while knowing him.  Not because you have seen him a few times in town.

I can't help it, I'm so attracted to this man. I wonder now if he was ever attracted to me at all, or knows how much I feel about him. I sense he feels something for me but now sure how much. I know I should leave it in the past but I can't seem to let go & it seems at the time when he approached me, he hadn't forgotten me all those years ago either...

He probably does fancy you, but really if he is happy in his marriage, would he really be wise to risk all that.  Both married guys and married women have whims where they see other more attractive people.  Those whims can destroy a perfectly good marriage.  Fair enough if your marriage has been going badly over a series of years it might be best to get out of it, but really in any other situation you need to think carefully before destroying your marriage and running off with what could be a very brief fling.

Want to get the opinion of other ladies who visit this web site?  Try my new forum .