Error 400
Hi,
I have tried to execute the following code (from samples) with correct username and API key but it returns an error 400, can you please advise if there's anything wrong here,
CODE:
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.Base64;
public class Main {
public static void main(String[] args) {
try {
String username = ""; // removed on purpose
String apiKey = ""; // removed on purpose
String originalInput = username + ":" + apiKey;
String encodedString = "Basic " + Base64.getEncoder().encodeToString(originalInput.getBytes());
String apiEndPoint = "http://api.scraping-bot.io/scrape/retail"; URL url = new URL(apiEndPoint); HttpURLConnection con = (HttpURLConnection) url.openConnection(); con.setRequestMethod("POST"); con.setRequestProperty("Content-Type", "application/json; charset=UTF-8"); con.setRequestProperty("Authorization", encodedString); String useChrome = "false";//set to "true" if you want to use headless chrome for javascript rendering String premiumProxy = "false";//set to "true" if you want to use premium proxies Unblock Amazon,Google,Rakuten String urlToScrape = "https://www.scraping-bot.io/example-ebay.html"; String proxyCountry = null;//allows you to choose a country proxy (example: proxyCountry:"FR") String waitForNetworkRequests = "false";//set to 'true' if you want to use 'networkidle2' String param = "{\"url\":\""+urlToScrape+"\","+ "\"options\":{"+ "\"useChrome\":"+useChrome+","+ "\"premiumProxy\":"+premiumProxy+","+ "\"proxyCountry\":"+proxyCountry+","+ "\"waitForNetworkRequests\":"+waitForNetworkRequests+","+ "}"+ "}"; con.setDoOutput(true); OutputStream out = con.getOutputStream(); out.write(param.getBytes()); out.flush(); out.close(); int status = con.getResponseCode(); System.out.println(status); BufferedReader in = new BufferedReader( new InputStreamReader(con.getInputStream())); String inputLine; StringBuilder content = new StringBuilder(); while ((inputLine = in.readLine()) != null) { content.append(inputLine); } String jsonResponse = content.toString(); System.out.println(jsonResponse); in.close(); con.disconnect(); } catch (Exception e) { System.out.println("An error occured while scraping:" + e); } }
}
ERROR:
An error occured while scraping:java.io.IOException: Server returned HTTP response code: 400 for URL: http://api.scraping-bot.io/scrape/retail
hmm that's strange, can you send screenshot of the full error?
Error 400 means 'Bas request' so there's likely something wrong with the headers.
What line does the error occur on?
400 is a bad request error, you probably have some link with a bad sign, so a bad link, with something invalid in it.