Earn 81,000 ($810.00)

Dont apply I have discussed it with a dev

Description:
I am looking for a skilled developer to build a scalable AI agent capable of scraping data from websites efficiently and reliably. The agent should handle dynamic content, pagination, and anti-scraping measures while ensuring data accuracy and scalability for large-scale projects.

Requirements:

Core Functionality:
Scrape structured and unstructured data from websites.
Handle dynamic content (e.g., JavaScript-rendered pages).
Manage pagination and multi-level navigation.
Bypass common anti-scraping measures (e.g., CAPTCHA, rate limiting).

Scalability:
Ability to scale across multiple websites simultaneously.
Support for distributed scraping (e.g., using cloud infrastructure).
Efficient resource management to handle large datasets.

Output:
Store scraped data in a structured format (e.g., JSON, CSV, or database).
Provide real-time progress tracking and error logging.

Tech Stack:
Preferred languages: Python (with libraries like BeautifulSoup, Scrapy, Selenium).
Optional: Integration with AI/ML models for data extraction and analysis.

Additional Features:
Customizable scraping rules for different websites.
User-friendly interface or API for configuration and monitoring.
Compliance with legal and ethical scraping practices (e.g., respecting robots.txt).