A few thoughts:
1. While I tend to come down on the side of "words are one thing, deeds are quite another" and the license that implies that we should all, and always, be judged not by what we say, but instead, by what we do, I have over time become much more aware of the corrosive effects of language. Effects that, aside from hate-speech, we may have grown used to and may only be minimally aware of, if at all.
2. In any case, being non-Japanese, it simply isn't my call to decide if the term is offensive or not.
3. I would point out, however, the obvious fact that "jap" happens to be the first three letters of your country's name and the ethnicity of it's residents, so it may simply be a verbal short-hand or abbreviation that is used casually and without bad intent. Citizens of Britain are, after all, widely-known and referred to as "Brits" and, aside from the term's use by members of the Irish Republican Army, I don't think is meant or taken as an insult or slur. Similarly, people of the Jewish faith have no problem being called "Jews" and universally refer to themselves as such.
This is not to brush aside or show a lack of respect for your feelings, fightclub85, but to suggest that not every such use of the term may be rooted in racism.
There is speech and there is behavior. While the former can be unclear, the latter never is. It is our actions -- not our thoughts or words -- that define us.
