Problem Description

Create APIs to enable LLM powered call answering, call forwarding, and voice question answering experience.

Acceptance Criteria

• Able to purchase a Twilio number (e.g. POST /number)
• Attach a LLM to the Twilio number (e.g. POST /number/agent/)
• Another user is able to call the Twilio number and talk to the LLM.
• The user can ask the agent to transfer the call, which would transfer the call to a different $PHONE_NUMBER. The call will not be dropped until the transfer is complete (the other side picks up). Music plays during transfer.
• The call is terminated by call server if silence persists more than 7 seconds
• LLM agent can use user-provided text to perform RAG to ground the response
• Given a list of phone numbers, agent call each of the number. If a human picks up, have LLM speak (using ElevenLabs voice API).
• Leave a voicemail if call isn't picked up using answering machine detection.
• Transcribe the entire call using OpenAI whisper and persist the call MP3 and transcript to S3
• Simple UI to expose these features

Final Acceptance Criteria

• github repo (will send over invite) for the code
• npm run dev should start the UI
• node call_center.js should start the backend server.
• dockerfile to deploy the call center.js
• End to end testing and code review

Technical Details

• NodeJS / Typescript perferred (no python unless absolutely necessary)
• Architecture: backend( node LLM agent server, and restful apis to manage the agent) and UI server. External APIs: OpenAI, S3, ElevanLabs, Twilio
• The chabot is powered by GPT3.5 and voice is generated by ElevanLabs in the streaming generation mode. Use elevenlabs streaming for synthesis to reduce latency (https://elevenlabs.io/docs/api-reference/streaming)
• Make LLM easy to switch out via using Vercel AI