Earn 4,500 ($45.00)

Hello,

I am seeking a proficient developer to create an advanced web scraping and monitoring system. The primary target of this project is Snapshot (https://snapshot.org/), a decentralized voting platform where multiple organizations host voting proposals.

The overarching goal of this project is to facilitate real-time monitoring of multiple organization pages on Snapshot for new proposals and to receive instant notifications via Discord whenever a new proposal is added to any of these organizations. I will be providing a list of organizations to be monitored separately. However, it is crucial that the developed solution has the flexibility for easy addition and removal of organizations from the monitoring list.

Here are the specific requirements in further detail:

1. Snapshot Scraper: The developed solution should have the ability to connect to Snapshot's GraphQL endpoint or leverage Scrapy to scrape the organization pages on Snapshot. Given that Snapshot is a dynamic website, the spider might have to use utilities like Splash or Selenium to render JavaScript or identify the API used by the website to load data.

2. Proposal Monitoring: The spider should have a mechanism to identify when a new proposal is added to any of the monitored organization pages. We will define what constitutes a "new" proposal at the inception of the project, ensuring there's no ambiguity during the execution phase.

3. Dynamic Organization List: The monitoring solution should have a feature to allow the list of monitored organizations to be easily modified. This means the capability to flexibly add or remove organizations should be a part of the tool without the need to modify the core code of the spider.

4. Automated Spider: The spider should be programmed to run autonomously at regular intervals. For this project, the desired frequency is every 10 minutes. This will ensure that we capture and analyze information as soon as it's available.

5. Discord Notifications: Upon identifying a new proposal, the spider should be capable of sending a notification to a predefined Discord channel using Discord webhooks. This notification should include comprehensive details about the new proposal, such as the title, URL, and the organization it belongs to.

6. Deployment Guidance: Part of the deliverables should include step-by-step guidance on how the script could be deployed and run continuously in different environments. This should cover a variety of deployment scenarios to ensure we have flexibility in how we use the solution.

The end deliverables for this project will include the complete, working Scrapy spider code, in addition to comprehensive instructions on how to set up, use, and modify the list of monitored organizations in the spider.

I invite you to consider this project and am eagerly looking forward to your proposals. Thank you for your time and consideration.

Best Regards,
Ara Borious