This expression is used when you are 'annoyed' or 'curious' as to why someone is looking at you. It has the nuance of "mind your own business," but it can be said with a joking manner among friends.