where's the alpha? scraping telegram with llms

i led a team building an end-to-end, ai-powered analytics tool that scrapes text data from social platforms (starting with telegram) to generate personalised crypto market and sentiment insights using llms.

the project won 1st place and the starkware award at cube summit 2024.

when a user links their accounts, the tool gathers their message data. it then applies conventional nlp techniques - named entity recognition (ner) and tf-idf - to extract meaningful features.

those features are turned into embeddings and used to cluster related messages. the clustered data is fed into a language model (gpt-3, gpt-4, or gemini) to generate the final market and sentiment summaries.

data was stored on aws (s3 and dynamodb). the longer-term plan was to refine the language model with reinforcement learning from human feedback (rlhf) by showing it examples of effective summaries, and to expand scraping to discord and twitter.