I am trying to display LLM generated messages as they are generated. LLM generation tends to be slow in general, so displaying the message slowly, line by line, is a UI trick most sites use so that the user doesn’t feel like they are waiting forever.
I am using a custom model, not OpenAI, so I can’t use the WeWeb OpenAI plugin.
What is the best way to approach displaying this streaming text in WeWeb?
My Stack: Supabase collections in weweb, plus API calls to my coded server (have full flexibility there)
Some things I’ve been trying:
[Trying to see if real streaming could work]
Supabase realtime tables + dynamic collection (if I manually update the collection data, the change doesn’t appear in my WeWeb app until I manually refresh the page or collection. Is this how it is supposed to work, or should the data refresh itself with the change?)
[Trying a simpler task; how would I animate lines appearing one by one with a delay?]
Conditional rendering: I can get a simple conditional render to work - eg return false and the element doesn’t display. However, it doesn’t seem to render with a delay when a simple timeout is put in the JS function
Is there an easy way to animate display from none → block? It isn’t listed in transitions and I tried putting custom css in the box, but it is getting removed when I try it in preview mode.
@khara I am currently trying to figure this out as well as we are running the OpenAI Assistant API, all models from AWS Bedrock and now also Gemini Pro from Google Cloud.
@weweb-team Is the stream=true which is available for most Chat endpoints of GPT-4, Claude, Cohere, Jurassic, Titan, Llama possible to do in WeWeb natively or does this streaming functionality need to be implemented manually through an NPM package, etc?
Right! It seems like something probably a lot of people might need right now? Fingers crossed that there’s already a fast way to do this in WeWeb, but wondering if it’s necessary to roll a custom component.
@carrano_dot_dev thanks! I’m actually wondering about the frontend display for dynamically updating the streamed data though, I have the server part already.
I haven’t been able to get WeWeb to automatically update supabase collections when the data is updated, and wondering if there is a way to do that? Or alternatively if there is a way to stream directly to a WeWeb component.
Is there a way to do this already in WeWeb that I’ve missed? (Dynamically display text/data as it is streamed in)
Or if not, is this on the roadmap already for the next month? If not, then will look into building a custom component, but was hoping to avoid learning vue!
I’d imagine you would just create a text variable and bind it to a text element on the frontend. Then you could run a workflow that updates that variable as the data from the API is streamed in.
Hmm ok - so far as I can tell, the WeWeb text won’t update correctly with the streaming text, it sort of updates all at once when the API call finishes.
There might be a way to read the filestream directly from WeWeb’s workflow output object, using JS, but I couldn’t find it in a first pass.
lol just realized supabase was not updating because I didn’t have realtime enabled. Realtime updates are working there, so going to just try streaming directly to supabase for now.
I also looked into creating a coded component to handle streaming as well. Setup was pretty smooth, but then realized you need to be on a higher tier plan to actually use the coded components.
Ok final update from me. For anyone else who needs a quick solution here, streaming directly to a supabase table with realtime updates works decently well in practice.
This does use more of the realtime message quota on the free tier, but supabase is quite generous with that allowance. I ended up chunking together some of the streamed bits to save on DB resources, so it isn’t hitting it constantly.
It would be nice to stream directly to the client, so curious to know if this is on the roadmap at all if anyone from @weweb-team sees this.
I am doing this on my app what I have done is created a text variable and then used WeWeb copilot to create JavaScript where the value i want displayed is iterated one word at a time with a 200 ms delay and added to the text variable.
Works really well, I’ve built a form app that feels like a conversation as you respond to questions
I use the OpenAI plugin to generate the response and then update a variable directly from the OpenAI output using a workflow. I update the database later based on another trigger so that the user doesn’t have to wait for the form to retrieve the OpenAI response, store to the data, retrieve the value, etc etc…
Fingers crossed that there’s already a fast way to do this in WeWeb, but wondering if it’s necessary to roll a custom component.
Much gratitude, many thanks! I’ve been overly focused on updating the collection itself.
what I have done is created a text variable and then used WeWeb copilot to create JavaScript where the value i want displayed is iterated one word at a time with a 200 ms delay and added to the text variable.