How to get images in boards such as swe_tnote.com to load

[lardee]

Hi All
I've managed to track down some of the old mag scans that I would like on a picture board.
Funny thing is that I can open the posts sometimes in my browser but the images do not open in it, the page simply is blank.
This may be something to do with my location (not in Japan) as if I open the page or image link inside a translate page it opens just fine and I think the image page opened ok for the first few times. Maybe the website has some form of protection against crawling or multiple saving??
Tried to get hold of a bulk downloader to crawl the site but no joy as yet. Where there is discussion of this online it seems to suggest that the script running for the site is not a usual board one. Not that I know much about scripts.
Anyway I'd really appreciate if someone could advise if it is possible to take images from there using a firefox plugin, a stand alone downloader or the like.
I looked at registering for the board site but not sure that will help.
Thanks in Advance,
LL

 

[lardee]

Just to show that I'm not just a leecher and not too lazy here's an update.
I've managed to find a few possibilities of how to scrape the data I want and so taking a crash course in HTML, python, and java.
Hoping that scrapy or something similar will work. Will update if I manage to get it working.

 

[lardee]

Still struggling .... proprietary scripts......darn

 

[lardee]

Well.......couldn't get the scrapers or spiders to work so used the brute force of TOR and lots of clicks. Low tech high labour-intensive solution kinda works......

 

[Ceewan]

Downloading picture by picture is the best way for imageboards. People used to be able to access the images seperately but that left a huge security hole so nowadays you need admin privledges to access that area (which is why crawlers and site-rippers fail...you don't have the right pass so you get a 403 error). Just right click on the image you want, (some image boards won't even let you do that and on others it isn't even necessary), choose view image and then when the image loads just save the image to your drive and it will download through your browser. I usually try view image first because sometimes an imageboard resizes the image and if you download straight from the html page you don't get the full size. If you are downloading a lot of images, (which can be time consuming), then experiment to find what works best. If it is something you really want then it is worth the effort.

 

[lardee]

Thanks very much for the advice Ceewan and for taking time to help me.

Perhaps the following might help another person.
I've worked hard on getting a good way to automate the manual process of looking and right-clicking and am now I'm nearly there, I have not been able to use scripting or scraping or spidering to get what I wanted from the image board I wanted. However, there are some things to help speed up a long process.

You will need the Tor version of Firefox, and a few addons. The popular proxy for Firefox, Hola, did not help much to get into the pages and open them succesfully and I noticed the site stopped me using the google translate as a kind of proxy to get in after a few days. I suppose they keep an eye on their logs!!

So I use Tor to allow the pages to load reliably (if at all!), so effectively Firefox with an onion to hop around the internet (like on the various films about hacking and cyber-heroes). This means I can access the index pages that are often banned from outside Japan (I suppose) and see if a page is what I want.
Then you can use the Firefox addons to automate some parts of the process. First get the tools added to Tor (see below) and open the index page you have found. Then use copy links 0.1.7.1, which will harvest the image links from the page into the clipboard. You can open the file in a spreadsheet program such as WPS Spreadsheet to sort the links and get what you want and not what you don't, A=Z sort is your friend....
Then make sure that Tor is still able to load a page without being blocked by the server, if not then do a re-route of the hops to the server.
Then use a multi-link opener, such as: Copy URLs Expert 2.3.1.1, which can open all the images in separate tabs. Let them open sequentially as this will increase your chances of getting them all at once - a good tab manager helps here as well as having 100 tabs open can be a challenge.
The tricky part for me still is how to save the images that are open in the new tabs........
With firefox this is easy, just use image picker 1.9.3.1 addon. This is neat and can be configured to offer all images from your open tabs as downloads. You see the images, check the sizes and pick those to download.
However, with Tor the addon does not seem to work, it gets the images into the selection screen but will not download them. It works with firefox perfectly fine just not with Tor! Not sure why not and not as yet able to get it to work. So I still have to click through each tab and use manual methods to finally get the files. It's a lot quicker though with sweetbo@rd pages of up to 100 images each.
Frustrating, but a little quicker than the old process.

Cheers
L

 

[lardee]

OK to get the images back it is possible to look in the TOR cache, so with the browser still open look for the cache, you can find it by typing about:cache
and you can then see where the files are. Such as here:
C:\Users\*********\Desktop\Tor Browser\Browser\TorBrowser\Data\Browser\Caches\profile.default\Cache\
Good luck!

 

[Ceewan]

Shit!......I just would have hunt and pecked!

By the way, don't use Hola....Hola very bad. See link in the news section.

 

[lardee]

Thanks again, I'll rid myself of Hola

 

[lardee]

One kinda useful thing I've found, using Tor you can specify the end point of the onion, the locale that you appear to be in.
To do this open up the torrc file and add a line at the end:
ExitNodes {JP}
if you want the server to think you are in Japan or {US} if you want the last hop to be from the USA or {GB}, etc
Helped a bit to get the last smidgens of data out. But the server increasingly rejected my requests for access to image pages over time. They are learning!
Managed to get ImagePicker to work with Tor now so all good. Just a little bit of work to transfer the links for individual images from/to the browser.
However, I think I have sucked out all that I wanted and only just worked out the best way to do it. Ho hum.....

 

[Ceewan]

ya know you probably could have found a working public proxy that would have donwe the job in half the time. not all japanese proxies work for that site but some from the neighboring regions might.

 

[lardee]

aye. true. and thanks again.
hindsight is a wonderful thing.
i think though a flash proxy thingie like tor may be better than going international. plus noone will be able to snoop.
what I have done is also proving it's worth on the odd image board, cough cough.....