yes, I looked back at the context where this word was used, and I personally didn't find it offensive. I didn't take it to mean that you were actually referring to anyone as infidels, but more that maybe your fictional klicky would view it that way. I guess it's just one of those subjects where you have to be careful because of current world events.
I think we all know that this has been playful banter, but the concern maybe is that it will begin to get out of hand if racial, ethnic or religious slurs are used, as some are offensive whether said playfully or not.