Skip to content
    Back to all Bounties

    Earn 50,000 ($500.00)

    Time Remainingdue 2 years ago
    Completed

    Use Browser-based Vision AI SDK + ChatGPT for Demo App that Modifies Prompt based on detected emotion

    scottjeezey
    scottjeezey
    Posted 2 years ago
    This Bounty has been completed!
    @scottjeezey's review of @CoderGautamYT
    5.0
    Average Rating
    Communication 5/5, Quality 5/5, Timeliness 5/5
    “This is my first time using Replit after hearing great things from an advisor. I posted this job originally in the 2nd half of January. I had two false starts with devs who picked it up and then flaked. I was not optimistic! Then Guatam picked it up, exuded confidence from his first message, and knocked out a protoype in what appeared to be a matter of hours. The project took a week to finish due to my delays in giving feedback. Gautam is A+ and truly an example of the best of Replit's potential, as was described to me by my advisor in January. Would love to work with Gautam again!”

    Bounty Description

    Problem Description

    I am interested in getting a POC demo application that modifies the prompt to Open AI's ChatGPT to the real-time detected emotional state of a user via a Javascript Vision AI SDK. In other words, imagine using the emotional state of the user as a feedback loop and input to the text that OpenAI generates. I would like for you to build this demo app using my company's alpha JavaScript SDK for Vision AI, which can be found here: Realeyes Experience Platform Javascript SDK - Getting Started Guide

    The app should work something like this:
    • User opens a webapp and grants permission to the Vision AI SDK for camera access. You can display the webcam live image and emotion telemetry on the web app screen for the purposes of this demo application.
    • User is presented with some sort of opening set of text displayed on the screen -- generated by the web app, or perhaps even by ChatGPT.
    • The user is prompted to make facial expressions -- at least inclusive of happy and sad.
    • When an emotional state is detected (e.g., a 10 second average recording of that value), a prompt is modified to ChatGPT reflecting that emotional state.
    • The next set of text returned from ChatGPT should be modified to reflect the measured emotional state of the user.
    • I'd like to see a dynamic looping of the feedback such that, for example, the the demo would show:

    1. Sad user emotion leads to sad text returned by ChatGPT
    2. Happy user emotion leads to happy text returned by ChatGPT
    3. Angry user emotion leads to angry text returned by ChatGPT

    In other words, it should be clear through the demo app that the prompts to ChatGPT are being dynamically modified based on the most recent detected emotion of the user.

    Acceptance Criteria

    • Utilization of the Realeyes Experience Platform JavaScript SDK that measures a users' emotional state off of webcam.
    • Ideally built using React Native so that it can run on mobile
    • At minimum I'd like to see a demo that leverages "happy" and "sad" emotion metrics to modify the ChatGPT prompt and subsequent output. A third emotion like "anger" would be great.
    • Incorporation of ChatGPT with automated prompt modification based on detected emotion, and showcasing the dynamic nature of it such that a new emotion can return an updated prompt and result in new text sentiment from ChatGPT
    • The code has to be clean and well documented
    • The UX in the browser web app needs to be minimal and clean
    • Bounty hunter must have experience building both with OpenAI and doing frontend programming
    • Hunter should respond quickly and meet deadline