Web Scraping API
Targeting websites not listed in Available targets list
You can use universal
parameter as your target and supply any URL you want, which will return the HTML of the targeted URL.
Code examples for targeting ip.smartproxy.com
website (with authorization details changed only):
import requests
url = "https://scrape.smartproxy.com/v1/tasks"
payload = {
"target": "universal",
"headless": "html",
"url": "https://ip.smartproxy.com"
}
headers = {
"Accept": "application/json",
"Content-Type": "application/json",
"Authorization": "Base64 encoded user:pass"
}
response = requests.request("POST", url, json=payload, headers=headers)
print(response.text)
<?php
require_once('vendor/autoload.php');
$client = new \GuzzleHttp\Client();
$response = $client->request('POST', 'https://scrape.smartproxy.com/v1/tasks', [
'body' => '{"target":"universal","headless":"html","url":"https://ip.smartproxy.com"}',
'headers' => [
'Accept' => 'application/json',
'Authorization' => 'Base64 encoded user:pass',
'Content-Type' => 'application/json',
],
]);
echo $response->getBody();
const options = {
method: 'POST',
headers: {
Accept: 'application/json',
'Content-Type': 'application/json',
Authorization: 'Base64 encoded user:pass'
},
body: JSON.stringify({target: 'universal', headless: 'html', url: 'https://ip.smartproxy.com'})
};
fetch('https://scrape.smartproxy.com/v1/tasks', options)
.then(response => response.json())
.then(response => console.log(response))
.catch(err => console.error(err));
curl --request POST \
--url https://scrape.smartproxy.com/v1/tasks \
--header 'Accept: application/json' \
--header 'Authorization: Base64 encoded user:pass' \
--header 'Content-Type: application/json' \
--data '
{
"target": "universal",
"headless": "html",
"url": "https://ip.smartproxy.com"
}
'
Result of the above code:
Once you have an active Web subscription, you will be able to check your generated proxy Username as well as set your desired Password in the Authentication method tab.
You can try sending a request via our API Playground by choosing your desired Parameters for the request. In this case, the Target will always be Universal.
You will also see an example of curl request generated on the right side of the screen.
Updated 4 months ago