Web Scraping API

Targeting websites not listed in Available targets list

You can use universal parameter as your target and supply any URL you want, which will return the HTML of the targeted URL.

Code examples for targeting ip.smartproxy.com website (with authorization details changed only):

import requests

url = "https://scraper-api.smartproxy.com/v2/scrape"

payload = {
    "target": "universal",
    "headless": "html",
    "url": "https://ip.smartproxy.com"
}
headers = {
    "Accept": "application/json",
    "Content-Type": "application/json",
    "Authorization": "Base64 encoded user:pass"
}

response = requests.request("POST", url, json=payload, headers=headers)

print(response.text)

<?php
require_once('vendor/autoload.php');

$client = new \GuzzleHttp\Client();

$response = $client->request('POST', 'https://scraper-api.smartproxy.com/v2/scrape', [
  'body' => '{"target":"universal","headless":"html","url":"https://ip.smartproxy.com"}',
  'headers' => [
    'Accept' => 'application/json',
    'Authorization' => 'Base64 encoded user:pass',
    'Content-Type' => 'application/json',
  ],
]);

echo $response->getBody();

const options = {
  method: 'POST',
  headers: {
    Accept: 'application/json',
    'Content-Type': 'application/json',
    Authorization: 'Base64 encoded user:pass'
  },
  body: JSON.stringify({target: 'universal', headless: 'html', url: 'https://ip.smartproxy.com'})
};

fetch('https://scraper-api.smartproxy.com/v2/scrape', options)
  .then(response => response.json())
  .then(response => console.log(response))
  .catch(err => console.error(err));

curl --request POST \
     --url https://scraper-api.smartproxy.com/v2/scrape \
     --header 'Accept: application/json' \
     --header 'Authorization: Base64 encoded user:pass' \
     --header 'Content-Type: application/json' \
     --data '
{
     "target": "universal",
     "headless": "html",
     "url": "https://ip.smartproxy.com"
}
'

Result of the above code:

Authentication

Once you have an active Web subscription, you'll be able to check your generated proxy Username as well as see or regenerate a new Password in the API Authentication tab.

Sending requests via dashboard

You can try sending a request via your dashboard in the Scraper section by selecting your Target from the drop-down menu and choosing your desired parameters for the request. Click More options to reveal additional functions.

You will also see the response and an example of a cURL request generated at the bottom if you select the Response or the Request tab respectively.

Smartproxy dashboard – the Scraper section

Postman

Import our Web Scraping API request collection to Postman:

Updated 17 days ago