Gimmeproxy was heavily updated recently, check this article: http://256cats.com/gimmeproxy-api-major-update/

Recently I needed a lot of free proxy servers for one project. There was another requirement: proxies had to be rotated.

That means I needed to change proxy after several requests and needed many. I couldn’t use the same proxy for more than 400 requests per hour, so I had to manage some kind of timeouts.

That’s easy, I thought: get list of proxies somewhere, check them and choose one at a time by random. But it turned out to be not so easy.

First of all, obtaining free proxy list in txt or csv format was not so FREE. In fact a lot of websites offer that kind of service. They allow to check some of their proxies and then try to sell you full updated list for $3-20 per month. Others have some subscription-based API.

I had no idea what I was going to pay for. Some proxies from the free list just didn’t work at all, some didn’t allow to connect to my target website, even when they were advertised as ‘checked’ 3 minutes ago. Some did work though.

So I still had to check every proxy and set timeouts.

Long story short, I decided to build my own free rotating proxy api. It’s RESTful and called GimmeProxy.com.

GimmeProxy.com

For now it’s simple: it scrapes public lists of proxies from different websites, checks them and returns you random one in JSON.

It’s free and you don’t need to register or anything. Just think up and use any string that will be your user id and you are good to go. You get new random proxy with each request to the API.

Also you can set timeout in seconds. During that time you won’t get the same proxy again. Or you can set it to 0, if you don’t care.

For example: http://gimmeproxy.com/api/get/8bb99df808d75d71ee1bdd9d5da5672a/?timeout=0

Here user id is 8bb99df808d75d71ee1bdd9d5da5672a and timeout is 0 (i.e. no timeout).

Example response: {“ip”:”123.225.106.40:80”, “country”:”usa”, “type”:”http”, “curl”:”http://123.225.106.40:80"}

Ip, country and type are self-describing and curl is _CURLOPT_PROXY_ option of Curl library.

PHP Curl Proxy example

It’s simple to use gimmeproxy with PHP.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
function getProxy() {
	$data = json_decode(file_get_contents('http://gimmeproxy.com/api/get/8bb99df808d75d71ee1bdd9e5d/?timeout=0'), 1);
	if(isset($data['error'])) { // there are no proxies left for this user-id and timeout
		echo $data['error']."\n";
	} 
	return isset($data['error']) ? false : $data['curl'];
}
 
function get($url) {
	$curlOptions = array(
		CURLOPT_CONNECTTIMEOUT => 25,
		CURLOPT_TIMEOUT => 25,
		CURLOPT_URL => $url,
		CURLOPT_SSL_VERIFYPEER => false,
		CURLOPT_SSL_VERIFYHOST => false,
		CURLOPT_FOLLOWLOCATION => true,
		CURLOPT_MAXREDIRS => 9,
		CURLOPT_RETURNTRANSFER => TRUE,
		CURLOPT_HEADER => 0,
		CURLOPT_USERAGENT => "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36",
		CURLINFO_HEADER_OUT  => true,
	);
	$curl = curl_init();
	curl_setopt_array($curl, $curlOptions);
	if($proxy = getProxy()) {
		echo 'set proxy '.$proxy."\n";
		curl_setopt($curl, CURLOPT_PROXY, $proxy);
	}
	$data = curl_exec($curl);
	curl_close($curl);
	return $data;
}
for($i = 0; $i < 5; $i++) {
	echo trim(get('http://icanhazip.com/'))."\n";
}

Node.js Curl Proxy example

At first install curl-request module.

install curlrequest```
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
 
Then create this file:
``` javascript
var curl = require('curlrequest');
 
var curlGet = function(url, proxy, callback) {
	var options = {
		url: url,
		retries: 5,
		headers: {
			'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36',
		},
		timeout: 15,
	};
 
	if(proxy) {
		options.proxy = proxy;
	}
 
	curl.request(options, function(err, res) {
		if(callback) callback(err, res);
	});
}
 
var getPage = function(url, callback) {
	curlGet('http://gimmeproxy.com/api/get/8bb99df808d75d71ee1bdd9e5d/?timeout=1', null, function(err, json) { // get new proxy
 
		var data = JSON.parse(json), proxy = null;
		if(!data || data.error) {
			console.log('Error:', data);
		} else {
			proxy = data.curl;
			console.log('Set proxy', proxy);
		}
		curlGet(url, proxy, callback);
	});
}
 
var test = function(i) {
	getPage('http://icanhazip.com/', function(err, res) {
		console.log(res);
		i++;
		if(i < 5) test(i);
	});
}
 
test(0);

If you are interested, please give it a try. Still some proxies might not work for your target servers, but you can always get another proxy.

Any comments and questions are welcome as usual.

Tech stack

This site was built with NodeJS, Express, Redis, Cheerio. I’ve used Redis as a database, especially it’s excellent Lua scripting feature.

Realtime stats was made with Socket.IO and Redis PUB/SUB.

To run all scripts I used PM2.