Web Scraping API

With our Web Scraping API, you can use the universal parameter as your target and supply any URL you want, which will return the HTML of the targeted URL.

You can integrate the API into your code, or test requests via your dashboard using templates or the playground.

Authentication

Once you have an active Web subscription, you'll be able to check your generated proxy Username as well as see or regenerate a new Password in the API Authentication tab.

Target any websites with the universal parameter via code

You can use universal parameter as your target and supply any URL you want, which will return the HTML of the targeted URL.

Code examples for targeting ip.smartproxy.com website (with authorization details changed only):

import requests

url = "https://scraper-api.smartproxy.com/v2/scrape"

payload = {
    "target": "universal",
    "headless": "html",
    "url": "https://ip.smartproxy.com"
}
headers = {
    "Accept": "application/json",
    "Content-Type": "application/json",
    "Authorization": "Base64 encoded user:pass"
}

response = requests.request("POST", url, json=payload, headers=headers)

print(response.text)

<?php
require_once('vendor/autoload.php');

$client = new \GuzzleHttp\Client();

$response = $client->request('POST', 'https://scraper-api.smartproxy.com/v2/scrape', [
  'body' => '{"target":"universal","headless":"html","url":"https://ip.smartproxy.com"}',
  'headers' => [
    'Accept' => 'application/json',
    'Authorization' => 'Base64 encoded user:pass',
    'Content-Type' => 'application/json',
  ],
]);

echo $response->getBody();

const options = {
  method: 'POST',
  headers: {
    Accept: 'application/json',
    'Content-Type': 'application/json',
    Authorization: 'Base64 encoded user:pass'
  },
  body: JSON.stringify({target: 'universal', headless: 'html', url: 'https://ip.smartproxy.com'})
};

fetch('https://scraper-api.smartproxy.com/v2/scrape', options)
  .then(response => response.json())
  .then(response => console.log(response))
  .catch(err => console.error(err));

curl --request POST \
     --url https://scraper-api.smartproxy.com/v2/scrape \
     --header 'Accept: application/json' \
     --header 'Authorization: Base64 encoded user:pass' \
     --header 'Content-Type: application/json' \
     --data '
{
     "target": "universal",
     "headless": "html",
     "url": "https://ip.smartproxy.com"
}
'

Result of the above code:

Sending requests via the dashboard using templates

You can try sending a request via your dashboard in the Scrapers section by creating a new project, inputting your URL, and customizing your desired parameters for the request.

1. Click Create new project.

2. Open the dropdowns to specify your custom parameters.

4. You will also see the response in HTML and in the form of a Live Preview, as well as an example of a cURL, Node, or Python request if you select the Response or the Request tab respectively.

• You can Copy or Export the Response in HTML format.
• You can Copy the Request in each available language.

Saving & scheduling scraping templates

You can save a specific template by clicking Save new scraper once you select your parameters.

• Once saved, it will appear in the Scrapers section and you can click it to start working on it again.
• Click Update scraper to update the template.

To create a Schedule, click Schedule while a template is selected, and a right-side menu will pop up.

• You can specify the scheduling's frequency, delivery method (email or webhook), and the format in JSON.

Sending requests via the dashboard using the playground

You can try sending a request via your dashboard in the Scraper section by selecting your target from the drop-down menu and choosing your desired parameters for the request. Click More options to reveal additional functions.

You will also see the response and an example of a cURL request generated at the bottom if you select the Response or the Request tab, respectively.

Postman

Import our Web Scraping API request collection to Postman: