So what I did is I started with one of my friend and went to his follower. Then I started adding some of his friend to follow him. I was expecting some people to follow me too. But I build a list of people whom I am following but few are following me. So I decided to pull out the plug on most of these people. So I do it.
Here's a simple way to do it.
1. First go to the person "X" profile.
2. On the right hand side if you see "Actions: message username" then that user is following you.
If not, than the person is not following you. So you can get rid of them.
There are many people who have developed small API code for this and put on the web for other's to use it.
Here's the one I use sometime (sorry, I find the first one simple and easy).
http://following.reallyneatapps.com/
Twit This!