Web Scraping API

Guide by Jeremy Larsen

Targeting websites not listed in Available targets list

You can use universal parameter as your target and supply any URL you want, which will return the HTML of the targeted URL.

Code examples for targeting ip.smartproxy.com website (with authorization details changed only):

import requests

url = "https://scrape.smartproxy.com/v1/tasks"

payload = {
    "target": "universal",
    "parse": False,
    "url": "https://ip.smartproxy.com"
}
headers = {
    "Accept": "application/json",
    "Content-Type": "application/json",
    "Authorization": "Base64 encoded user:pass"
}

response = requests.request("POST", url, json=payload, headers=headers)

print(response.text)
<?php
require_once('vendor/autoload.php');

$client = new \GuzzleHttp\Client();

$response = $client->request('POST', 'https://scrape.smartproxy.com/v1/tasks', [
  'body' => '{"target":"universal","parse":false,"url":"https://ip.smartproxy.com"}',
  'headers' => [
    'Accept' => 'application/json',
    'Authorization' => 'Base64 encoded user:pass',
    'Content-Type' => 'application/json',
  ],
]);

echo $response->getBody();
const options = {
  method: 'POST',
  headers: {
    Accept: 'application/json',
    'Content-Type': 'application/json',
    Authorization: 'Base64 encoded user:pass'
  },
  body: JSON.stringify({target: 'universal', parse: false, url: 'https://ip.smartproxy.com'})
};

fetch('https://scrape.smartproxy.com/v1/tasks', options)
  .then(response => response.json())
  .then(response => console.log(response))
  .catch(err => console.error(err));
curl --request POST \
     --url https://scrape.smartproxy.com/v1/tasks \
     --header 'Accept: application/json' \
     --header 'Authorization: Base64 encoded user:pass' \
     --header 'Content-Type: application/json' \
     --data '
{
     "target": "universal",
     "parse": false,
     "url": "https://ip.smartproxy.com"
}
'

Result of the above code:

423423

Once you have an active Web Scraping API subscription, you can try sending a request right from the dashboard Web > Authentication method tab simply by entering your username, password and clicking on Generate. You will also see an example of curl request generated right below your entered user:pass.

Note that this is only an example with preset values to get you on the right track for forming your own request, meaning you will not be able to change the request values in the dashboard itself – that will have to be done in your code.

18441844

Check out this recipe for a step by step explanation of how to implement Web Scraping API