Hey whatsup fellas. After opening the courses and scripts sections im updating this blog with something new and different. How many times you wanted to make a private gallery from many different tumblr profiles to watch offline? Maybe zero, but with this knowledge you will be able to do it with just 1 line of code!

So what it is going to do, is using curl to mass scrape and paginate a tumblr profile, so you will get a list of URLs that will be processed with a while loop inside the curl, and then saving it on the folder you are running this command. But first…

You might need to install cURL in your server, dont worry is easy:

sudo apt-get install curl

And if this doesnt works, try doing an apt-get update and then:

sudo apt-get install libcurl3 php5-curl

Then just like that, type: curl -h  to see the help, if it display now you have it installed.

This is the main command you have to type inside the folder where you want to put all the downloaded images:

curl http://concept-art.tumblr.com/page/[1-7] | grep -o ‘src=”[^”]*.[png-jpg]”‘ | cut -d\” -f2 | while read l; do curl “$l” -o “${l##*/}”; done

Lets explain the code a little bit: the curl command is requesting the URL http://concept-art.tumblr.com/page/1 starting with page 1, then when we put in [ ] and separate a bigger number at the right with a dash – , we will be able to request multiple pages doing a pagination. Then when you add | means that it is separating the commands to run after, so grep is going to do a search for src attributes and only for those with termination on [png-jpg] or you can add more doing -gif inside the [ ]. Then the cut  command will collect the names, but what it is going to make download eveything will be the while loop, doing a curl on the found urls for the png or jpg images.

So make sure to add the while loop to the end for downloading, if you only want to see the images grabbed using the following command:

curl http://concept-art.tumblr.com/page/[1-7] | grep -o ‘src=”[^”]*.[png-jpg]”‘ | cut -d\” -f2

Here is a video that I recorded in order to show you how to do it: