First things first.
To successfully learn ANY language you NEED to have a desire to learn, a strong one, which means not letting other things "get in the way", that is just an excuse people use.
Meeting Japanese is not hard, but they are shy to speak English which is why they stick in their own circles. I have only been back in Vancouver for 2 years and I have a met a TON of Japanese students (well over 50) because if you talk to them, they will attempt to talk back. Some may speak English well but just don't think they do.
Back to the language learning. It's not hard to learn a language. I learned Korean strictly by hanging out with Koreans and having a Korean GF (for approx. 1.5 years in Vancouver, not Korea). By the time 2010 rolled around and I decided to live there for a year I only picked up slang and obscure terms that I probably wouldn't have heard my friends say. The same goes for Japanese. While I have taken the first entry level university course and passed with an A, having a Japanese GF and many Japanese friends and just listening to them and asking questions is the best way to learn.
Here are some other tips I can give you:
Buy an Android phone and download "Simeji" and use that to communicate with your friends as much as you can so you can practice using Japanese even when you aren't with them. If you aren't comfortable with using a layout entirely in Japanese you can use "Jellybean" but that is very limited in terms of using kanji etc. if you are interested in that and it's still forcing you to use romaji to write Japanese which isn't the correct way to learn.
Watch Japanese movies with English subtitles OR English movies with Japanese subtitles. You'll be surprised what you can pick up, every little bit helps.
Buy a book to learn Japanese, which was already mentioned before. With the age of the internet it's just a click away if they don't sell it locally.
You already do exchanges, which is by far the most effective way.
