Earn 19,800 ($198.00)

We are seeking a skilled DevOps Engineer to design and implement an automated web crawling system using Screaming Frog and cloud technologies. The system should dynamically create and manage virtual machines (VMs) for each crawl request, run Screaming Frog in headless mode, and upload crawl results to AWS S3.

Key Responsibilities:

• Develop a scalable system that creates VMs on-demand for each crawl request
• Automate the installation and configuration of Screaming Frog on the VMs
• Implement APIs for managing crawl projects (create, update, stop, pause, restart)
• Ensure efficient resource management by terminating VMs after crawl completion
• Set up monitoring and logging for performance optimization and troubleshooting

Required Skills:

• Strong experience with Google Cloud Platform (GCP) and AWS
• Proficiency in scripting languages (e.g., Bash, Python) for automation
• Knowledge of containerization technologies (e.g., Docker, Kubernetes)
• Familiarity with web crawling tools, preferably Screaming Frog
• Experience with API design and implementation
• Understanding of SEO principles and web crawling best practices

If you are a problem solver with a passion for automation and optimizing web crawling processes, we'd love to hear from you!

To apply, please add from Discord and share what you think about synincing Derby (embedded, no security on files) to 3rd party database in your first message to proof that you have read it.