Back to all Bounties
Earn 363,600 ($3,636.00)
due 1 year ago
Completed
Playground for local models
natfriedman
Details
Applications
20
Discussion
Accepted Application
0xhuda
Accepted
Remote inference demo: https://playground.0xhuda.repl.co/
Tweet with local inference demo: https://twitter.com/iamzainhuda/status/1619058598352674817?s=20
Hi Nat!
I've created a demo for you to try out (remote inferencing only) and also I've included a tweet where I go over local inferencing as well (loading hugging face models locally) and it extends to any text-generation model that is available on the platform (works with any T5 finetune you upload there)
I built it with flask and react with all logic, inference, and such lies on the server side to make it easy to change as needed and as well as extend to future providers or other ways to load models locally. The front end also dynamically displays parameters available for different models (cohere generation params and ranges differ from huggingface ones).
The following features are implemented in the demo:
+ Inferencing of remote models (Cohere is chosen as the provider, can be extended easily to other providers)
+ Model selection
+ Generation parameter selection
+ Undo/Regenerate buttons
+ Save/Load presets
+ T5 models locally (view the twitter link for a demonstration of this one)
I don't have much other public work but at facebook I launched a all-in-one performance profiling service that collects metrics, traces, and computes GPU analyses on over 8000 training jobs a day on Facebooks training cluster. Used by over 300 MAU.
https://pytorch.org/blog/performance-debugging-of-production-pytorch-models-at-meta/
Thanks for your time and consideration!