Hi Becky
You say "but then he wont speak to me, and then will again". This is a standard thing that men do, because men find that if they act too eager women go off them. So they go hot and cold all the time to make sure the women fall for them.  It would also be too much if he just pursued you constantly, it would be like stalking you.

You say "He also remembers nearly everything i have said", he is obviously very interested in what you say and hence must be very attracted to you if he takes the time to remember even minor details.

You say "he touches me alot". Again men don't touch women they don't find attractive.

You say "i really want to be with him, but i dont know how he feels or how to go about it" I am not sure what circumstances you meet him in (e.g. work, supermarket). But it sounds as though, you go out for a drink with him. So there are several ways to go about it
- Pretend you are a bit drunk (but don't get drunk because you won't remember anything). That way you can do all sorts like lean on him as you are sitting down or walking down the street. But if you touch him too much, he might think it is the green light for sex. If he rejects you, you can just apologise for being a bit tipsy.
- Throw some hints that you would like to go to a concert or the cinema. Not obvious hints, just bring it into the conversation "I am thinking about going to the concert, but none of my friends are available to go".

I can't really guarantee that the above things will work, because it depends on the situation you are in and depends on your execution at the time, as well as many other factors.

You just have to make yourself approachable without looking desperate and hopefully he will then fall for you.