should i scrape the soybooru?

[yunglimabean]

my motivation is that i am trans btw the sheer autism must be preserved and i have the storage or something, also the current shimmie scraper doesn't scrape comments from what i've heard so a custom scraper isn't too hard to make because all i gotta do is parse the dom, grab the image url and iterate through all the comments
as of writing there are 74092 submissions on the 'ru so i think that's a pretty large number but not large enough where it can't all be scraped within a day

 

[raepson industries .ltc]

didnt read but yeah sure do it

 

[ILoveVesna]

Wouldn’t it take a shit ton of storage doe?

 

[Nihilma]

Wouldn’t it take a shit ton of storage doe?

Yeah like how tf do you plan on storing allat ?

 

[ILoveVesna]

Yeah like how tf do you plan on storing allat ?

And upkeep for 109 boorutards would probably be expensive too. Where would you get funding for the storage?

 

[yunglimabean]

Wouldn’t it take a shit ton of storage doe?

assuming every submission is 4mb on average it would take up 296 gb, but like i said i have the storage to pull off this scrape

Yeah like how tf do you plan on storing allat ?

i have a 1tb drive i don't even use
also for every submission i will store metadata about it in a json format

 

[yunglimabean]

once this scrape is completed i'll upload it on archive.org for generations to come to witness the pure unfiltered autism

 

[Ballos]

How do you plan on getting past the McChallenge?

 

[yunglimabean]

How do you plan on getting past the McChallenge?

by solving the captcha and applying the cookie into the scraper

 

[.is refugee]

Yeah. No reason not to. Data Hoarding and the act of preserving something is one of the most respectable feats someone can do

 

[Hydrosophy]

over 70K images? I remember scraping the whole original booru when it had around 9K and thought I was mad. I don't see why not.
it's a shame I lost that folder though, it guaranteed a victory in every 'duel plus it had some jaks that I think are lost on the current booru

 

[yunglimabean]

over 70K images? I remember scraping the whole original booru when it had around 9K and thought I was mad. I don't see why not.
it's a shame I lost that folder though, it guaranteed a victory in every 'duel plus it had some jaks that I think are lost on the current booru

there could be deleted images too within those 70k images so i will need to handle that in the event i find an id that does not work

 

[yunglimabean]

Yeah. No reason not to. Data Hoarding and the act of preserving something is one of the most respectable feats someone can do

gen beta must see all the posts with the tag HAMAS_2_IS_A_OBESE_MUTT_SHITSKIN obviously

 

[yunglimabean]

i've started scraping the 'ru now

 

[yunglimabean]

so far i've scraped 25444 entries in 10 hours
48736‬ more to go

 

[basil]

so far i've scraped 25444 entries in 10 hours
48736‬ more to go

can't never trust a robot, i hope your computer is safe..

 

[yunglimabean]

can't never trust a robot, i hope your computer is safe..

why would my computer be in danger doe

 

[Lism]

my motivation is that i am trans btw the sheer autism must be preserved and i have the storage or something, also the current shimmie scraper doesn't scrape comments from what i've heard so a custom scraper isn't too hard to make because all i gotta do is parse the dom, grab the image url and iterate through all the comments
as of writing there are 74092 submissions on the 'ru so i think that's a pretty large number but not large enough where it can't all be scraped within a day

It's hard to scrape the 'ru independently these days because it has captcha. If you can then I might ask you to make a scraper for only my posts on there as well.

Will the data scaped preserve the tags and uploader of the images?

 

[yunglimabean]

It's hard to scrape the 'ru independently these days because it has captcha.

i just put a solved mcchallenge cookie into my scraper and it can last for a day

Will the data scaped preserve the tags and uploader of the images?

yes, including comments

 

[yunglimabean]

also the scraping process is almost complete i only need to scrape 2000 more images