Real-time Requests

Web Scraping API Real-time Requests

Introduction

The real-time integration puts the entire job request into one URL, which is formed by the API.

Post Endpoint:

scraper-api.smartproxy.com/v2/scrape

Examples
https://scraper-api.smartproxy.com/v2/scrape?target=google_search&query=world&domain=com&access_token=pass2021
curl -u username:password 'https://scraper-api.smartproxy.com/v2/scrape' -H "Content-Type: application/json" -d '{"target": "google_search", "domain": "com", "query": "world"}'
<?php 

$username = "SPusername";
$password = "SPpassword";

$search = [
    'target' => 'google_search',
    'domain' => 'com',
    'query' => 'world',
    'parse' => true
];

$ch = curl_init();

$headers[] = 'Content-Type: application/json';

$options = [
   CURLOPT_URL => 'https://scraper-api.smartproxy.com/v2/scrape',
   CURLOPT_USERPWD => sprintf('%s:%s', $username, $password),
   CURLOPT_POSTFIELDS => json_encode($search),
   CURLOPT_RETURNTRANSFER => 1,
   CURLOPT_ENCODING => 'gzip, deflate',
   CURLOPT_HTTPHEADER => $headers,
   CURLOPT_SSL_VERIFYPEER => false,
   CURLOPT_SSL_VERIFYHOST => false
];
curl_setopt_array($ch, $options);

$result = curl_exec($ch);
if (curl_errno($ch)) {
    echo 'Error:' . curl_error($ch);
}
curl_close($ch);

$result = json_decode($result);
var_dump($result);

 ?>
import requests

headers = {
    'Content-Type': 'application/json'
}

task_params = {
    'target': 'google_search',
    'domain': 'com',
    'query': 'world',
    'parse': True
}

username = 'SPuserame'
password = 'SPpassword'
  
response = requests.post(
    'https://scraper-api.smartproxy.com/v2/scrape',
    headers = headers,
    json = task_params,
    auth = (username, password)
)
print(response.text)

Quick Start

  1. Send a query. To specify it, you can add parameters.
    • You need to post query parameters the same way you post JSON ones.
    • Don't forget to input your credentials for authentication (tokens). How to find it?
  2. The Web Scraping API retrieves the content you need.
  3. The data should come back with the HTTP status code 200, and it should be parsed in JSON format or contain raw HTML.

📘

Keep an Open Connection

  • If the connection is closed before the job is completed, the data is lost.
  • The timeout limit for open connections is 150 seconds. In a rare case of a heavy load, we may not be able to get the data to you.
  • You can collect data using the same connection and get an immediate response.

Support

Need help or just want to say hello? Our customer support is available 24/7. You can also reach us anytime via email at [email protected].

Feedback

Can't find what you're looking for? Request an article!
Have feedback? Share your thoughts on how we can improve.