Proxies with Python 'Requests' module

You can refer to the proxy documentation here.

If you need to use a proxy, you can configure individual requests with the proxies argument to any Requests method:

import requests

proxies = {
  "http": "http://10.10.1.10:3128",
  "https": "https://10.10.1.10:1080",
}

requests.get("http://example.org", proxies=proxies)

To use HTTP basic authentication with your proxy, use the http://user:[email protected]/ syntax:

proxies = {
    "http": "http://user:[email protected]:3128/"
}

I have found that urllib has some really good code to pick up the system's proxy settings and they happen to be in the correct form to use directly. You can use this like:

import urllib

...
r = requests.get('http://example.org', proxies=urllib.request.getproxies())

It works really well and urllib knows about getting Mac OS X and Windows settings as well.

The accepted answer was a good start for me, but I kept getting the following error:

AssertionError: Not supported proxy scheme None

The fix to this was to specify the http:// in the proxy URL, thus:

http_proxy  = "http://194.62.145.248:8080"
https_proxy = "https://194.62.145.248:8080"
ftp_proxy   = "10.10.1.10:3128"

proxyDict = {
              "http"  : http_proxy,
              "https" : https_proxy,
              "ftp"   : ftp_proxy
            }

I'd be interested as to why the original works for some people, but not me.

The main answer is now updated to reflect this :)

If you'd like to persisist cookies and session data, you'd best do it like this:

import requests

proxies = {
    'http': 'http://user:[email protected]:3128',
    'https': 'https://user:[email protected]:3128',
}

# Create the session and set the proxies.
s = requests.Session()
s.proxies = proxies

# Make the HTTP request through the session.
r = s.get('http://www.showmemyip.com/')

I like:

import os
import requests

os.environ['HTTP_PROXY'] = os.environ['http_proxy'] = 'http://http-connect-proxy:3128/'
os.environ['HTTPS_PROXY'] = os.environ['https_proxy'] = 'http://http-connect-proxy:3128/'
os.environ['NO_PROXY'] = os.environ['no_proxy'] = '127.0.0.1,localhost,.local'

r = requests.get('https://example.com') # , verify=False

The documentation gives a very clear example of the proxies usage:

import requests

proxies = {
  'http': 'http://10.10.1.10:3128',
  'https': 'http://10.10.1.10:1080',
}

requests.get('http://example.org', proxies=proxies)

What isn't documented, however, is the fact that you can even configure proxies for individual URLs, even if the schema is the same! This comes in handy when you want to use different proxies for different websites you wish to scrape.

proxies = {
  'http://example.org': 'http://10.10.1.10:3128',
  'http://something.test': 'http://10.10.1.10:1080',
}

requests.get('http://something.test/some/url', proxies=proxies)

Additionally, requests.get essentially uses the requests.Session under the hood, so if you need more control, use it directly:

import requests

proxies = {
  'http': 'http://10.10.1.10:3128',
  'https': 'http://10.10.1.10:1080',
}
session = requests.Session()
session.proxies.update(proxies)

session.get('http://example.org')

I use it to set a fallback (a default proxy) that handles all traffic that doesn't match the schemas/URLs specified in the dictionary:

import requests

proxies = {
  'http': 'http://10.10.1.10:3128',
  'https': 'http://10.10.1.10:1080',
}
session = requests.Session()
session.proxies.setdefault('http', 'http://127.0.0.1:9009')
session.proxies.update(proxies)

session.get('http://example.org')

I just made a proxy grabber and also can connect with the same grabbed proxy without any input.

Here it is:

#Import Modules

from termcolor import colored
from selenium import webdriver
import requests
import os
import sys
import time

#Proxy Grab

options = webdriver.ChromeOptions()
options.add_argument('headless')
driver = webdriver.Chrome(chrome_options=options)
driver.get("https://www.sslproxies.org/")
tbody = driver.find_element_by_tag_name("tbody")
cell = tbody.find_elements_by_tag_name("tr")
for column in cell:

    column = column.text.split(" ")
    print(colored(column[0] + ":" + column[1], 'yellow'))

driver.quit()
print("")

os.system('clear')
os.system('cls')

# Proxy Connection

print(colored('Getting Proxies from grabber...', 'green'))
time.sleep(2)
os.system('clear')
os.system('cls')
proxy = {"http": "http://" + column[0] + ":" + column[1]}
url = 'https://mobile.facebook.com/login'
r = requests.get(url, proxies=proxy)
print("")
print(colored('Connecting using proxy', 'green'))
print("")
sts = r.status_code

Here is my basic class in Python for the Requests module with some proxy configurations and stopwatch!

import requests
import time

class BaseCheck():
    def __init__(self, url):
        self.http_proxy  = "http://user:pw@proxy:8080"
        self.https_proxy = "http://user:pw@proxy:8080"
        self.ftp_proxy   = "http://user:pw@proxy:8080"
        self.proxyDict = {
                      "http" : self.http_proxy,
                      "https": self.https_proxy,
                      "ftp"  : self.ftp_proxy
                    }
        self.url = url

        def makearr(tsteps):
            global stemps
            global steps
            stemps = {}
            for step in tsteps:
                stemps[step] = { 'start': 0, 'end': 0 }
            steps = tsteps

        makearr(['init', 'check'])

        def starttime(typ = ""):
            for stemp in stemps:
                if typ == "":
                    stemps[stemp]['start'] = time.time()
                else:
                    stemps[stemp][typ] = time.time()

        starttime()

    def __str__(self):
        return str(self.url)

    def getrequests(self):
        g = requests.get(self.url, proxies=self.proxyDict)
        print g.status_code
        print g.content
        print self.url
        stemps['init']['end'] = time.time()
        #print stemps['init']['end'] - stemps['init']['start']
        x = stemps['init']['end'] - stemps['init']['start']
        print x

test = BaseCheck(url='http://google.com')
test.getrequests()

It was already tested, and the following code works. It needs to use HTTPProxyAuth.

import requests
from requests.auth import HTTPProxyAuth


USE_PROXY = True
proxy_user = "aaa"
proxy_password = "bbb"
http_proxy = "http://your_proxy_server:8080"
https_proxy = "http://your_proxy_server:8080"
proxies = {
    "http": http_proxy,
    "https": https_proxy
}

def test(name):
    print(f'Hi, {name}')  # Press Ctrl + F8 to toggle the breakpoint.
    # Create the session and set the proxies.
    session = requests.Session()
    if USE_PROXY:
        session.trust_env = False
        session.proxies = proxies
        session.auth = HTTPProxyAuth(proxy_user, proxy_password)

    r = session.get('https://www.stackoverflow.com')
    print(r.status_code)

if __name__ == '__main__':
    test('aaa')

Here is a wrapper class that simplifies scraping proxies and then making an HTTP POST or GET:

ProxyRequests

I share some code how to fetch proxies from the site "https://free-proxy-list.net" and store data to a file compatible with tools like "Elite Proxy Switcher" (format IP address:PORT):

PROXY_UPDATER - get free proxies from https://free-proxy-list.net/

from lxml.html import fromstring
import requests
from itertools import cycle
import traceback
import re

###################### Find proxies #########################################
def get_proxies():
    url = 'https://free-proxy-list.net/'
    response = requests.get(url)
    parser = fromstring(response.text)
    proxies = set()
    for i in parser.xpath('//tbody/tr')[:299]: # 299 proxies maximum
        proxy = ":".join([i.xpath('.//td[1]/text()')
                [0], i.xpath('.//td[2]/text()')[0]])
        proxies.add(proxy)
    return proxies


###################### Write to file in format IP address:PORT ######################
try:
    proxies = get_proxies()
    f = open('proxy_list.txt', 'w')
    for proxy in proxies:
        f.write(proxy + '\n')
    f.close()
    print ("DONE")
except:
    print ("MAJOR ERROR")