Web Scraping API

Targeting websites not listed in Available targets list

You can use universal parameter as your target and supply any URL you want, which will return the HTML of the targeted URL.

Code examples for targeting ip.smartproxy.com website (with authorization details changed only):

import requests

url = "https://scrape.smartproxy.com/v1/tasks"

payload = {
    "target": "universal",
    "headless": "html",
    "url": "https://ip.smartproxy.com"
}
headers = {
    "Accept": "application/json",
    "Content-Type": "application/json",
    "Authorization": "Base64 encoded user:pass"
}

response = requests.request("POST", url, json=payload, headers=headers)

print(response.text)

<?php
require_once('vendor/autoload.php');

$client = new \GuzzleHttp\Client();

$response = $client->request('POST', 'https://scrape.smartproxy.com/v1/tasks', [
  'body' => '{"target":"universal","headless":"html","url":"https://ip.smartproxy.com"}',
  'headers' => [
    'Accept' => 'application/json',
    'Authorization' => 'Base64 encoded user:pass',
    'Content-Type' => 'application/json',
  ],
]);

echo $response->getBody();

const options = {
  method: 'POST',
  headers: {
    Accept: 'application/json',
    'Content-Type': 'application/json',
    Authorization: 'Base64 encoded user:pass'
  },
  body: JSON.stringify({target: 'universal', headless: 'html', url: 'https://ip.smartproxy.com'})
};

fetch('https://scrape.smartproxy.com/v1/tasks', options)
  .then(response => response.json())
  .then(response => console.log(response))
  .catch(err => console.error(err));

curl --request POST \
     --url https://scrape.smartproxy.com/v1/tasks \
     --header 'Accept: application/json' \
     --header 'Authorization: Base64 encoded user:pass' \
     --header 'Content-Type: application/json' \
     --data '
{
     "target": "universal",
     "headless": "html",
     "url": "https://ip.smartproxy.com"
}
'

Result of the above code:

Once you have an active Web subscription, you will be able to check your generated proxy Username as well as set your desired Password in the Authentication method tab.

You can try sending a request via our API Playground by choosing your desired Parameters for the request. In this case, the Target will always be Universal.
You will also see an example of curl request generated on the right side of the screen.

Looking for a custom built solution?

Updated 4 months ago