Theres always people that wont understand it, and they are entitled to that. People worrying about it being a waste of money, sure thats fine too. They'd be wasting their money on something themselves as well probably in your eyes.
The thing that is the problem, is people who would be so insulting to make fun of you because of your hobby, calling it childish or such. But people who do such things, tend to be very insecure themselves and "need" to make fun of what they percieve to be a weakness of others in order to feel better about themselves.
I wouldnt be surprised if some of the most vocal people like that would secretly be jalous of the hobby, but are affraid to admit it.