If you like crawling & scraping and experienced on Node.js - let's discuss! Discord ID: yigitkonur

Problem Description

We need a system that automates web searches from a search engine by using Cloudflare's Browser Rendering API. The goal is to create a scalable solution that efficiently retrieves search results while mimicking human behavior to avoid triggering search engine's security measures. The system should manage sessions using the Cloudflare Browser Rendering API, log into account, and make search requests based on jobs from a queue. Each session's Browser Rendering API worker should take a job from the queue, send a search query to search engine, and retrieve the first 100 search results.

Acceptance Criteria

1. The system should manage sessions using the Cloudflare Browser Rendering API, logging into account and making search requests.
2. A job queue should be implemented to continuously feed search queries to the system. Each session's Browser Rendering API worker should take a job from the queue and send a search query to search engine.
3. The system should retrieve the first 100 search results for each query and store them in a structured format (e.g., JSON or CSV) for further processing.
4. The system should mimic human behavior by incorporating random delays between searches and handling CAPTCHAs that may appear during the process.
5. The system should be optimized to work within the current limits of the Cloudflare Browser Rendering API, which allows opening 2 new browsers per minute per account, resulting in a maximum of 120 browsers per hour or 5,760 browsers per day.
6. A mechanism should be implemented to keep browser instances alive by sending dev tools commands at least once every 60 seconds to prevent automatic termination, utilizing the "keep alive" option provided by the API.
7. The system should have a scalable architecture that can handle up to 1000 concurrent sessions and process a high volume of search queries efficiently.
8. Comprehensive documentation and instructions should be provided for setting up and running the system, including details on configuring the job queue and handling the pricing model needs.
9. The implementation of the job queue and session management using the Cloudflare Browser Rendering API is mandatory.