Streaming from OpenAI into a div?

Bishop

9 months ago

Wonder if anyone has done that

.

Bishop

9 months ago

It looks like XANO cannot do it

, but maybe some other real time database can

. Anyone know

? Unfortunately built this all on XANO

.

yoelfdz

9 months ago

Hey Bishop

, I use Flowise as a backend for everything related AI

, it supports streaming

Andreas Møller

9 months ago

We don

't support streaming responses from an API yet

, but you can do it with a custom action

, which is how we made the AI chatbot in the editor

.

@Andreas Møller

We don

't support streaming responses from an API yet

, but you can do it with a custom action

, which is how we made the AI chatbot in the editor

.

yoelfdz

9 months ago

Could you share that chatbot as a component

? many of us are building AI SaaS

, that would help a lot

. I embedded a chatbot from Flowise but currently having some issues due to that

, need to build a chatbot like that with streaming capabilities

yoelfdz

9 months ago

@Andreas Møller

Bishop

9 months ago

@

I think we can grab the code on the website

?

Bishop

9 months ago

And put it into a package

.

🙂

I

'm just looking at this now

. You might also want to look at groq

, it is so damn fast you don

't need streaming

yoelfdz

9 months ago

I think we would be missing the most important part

, the custom action that makes it possible to stream the response

. I have no interest on the design

, just want to see how they

've built it

Actually I am doing that right now

, a bit stuck

Andreas Møller

9 months ago

I am chatting with buildship about building a package for their new assistant feature

:

https://buildship.com/assistant-api

🔥2

The problem with the current one we use is that is its built specifically for kapa

.ai and it is difficult to make it easily configurable

yoelfdz

9 months ago

Cool

! Will it be a complete chatbot with the logic of which messages to show

(question

/answer

/question

.

) or only the code for streaminf w

/ buidship as a package

?

Bishop

9 months ago

Ah I hear you

. I

'm sure I can muddle through creating it but

@Andreas Møller

if you have the custom action code handy that would be really great to share with us

. It will definitely take me some time to figure out

.

@Andreas Møller

The problem with the current one we use is that is its built specifically for kapa

.ai and it is difficult to make it easily configurable

yoelfdz

9 months ago

Got u

, I

'm actually using Flowise for this

, but having something to analyse and learn from to build ours would be helpful

Andreas Møller

9 months ago

It will be a complete package I think

.

💪1

yoelfdz

9 months ago

Thanks for that

🙂

Andreas Møller

9 months ago

We might be able to separate the UI from the backend integration

, I am not sure

. If not you can just clone the package and make one for flowise

❤️1

Bishop

9 months ago

My question is this

: is there any code for the chatbot you demonstrated that I can see

(I can clone it

? where will I find it

?

)

i can adapt it relatively easily i just don

't want to start from scratch

@Andreas Møller

We might be able to separate the UI from the backend integration

, I am not sure

. If not you can just clone the package and make one for flowise

yoelfdz

9 months ago

Of course

, once having the one you are talking about I should be able to replicate it

Bishop

9 months ago

i think i get it

. here it is maybe just clone it and look at the code

:

https://gpt.toddle.site/

yoelfdz

9 months ago

That

's a preview url

, do you have the project to see on the editor

? Does that one support streaming

?

Andreas Møller

9 months ago

That one does not

yoelfdz

9 months ago

Ok

, will have to wait for the package then

Thanks for the update Andreas

Andreas Møller

9 months ago

👍

Bishop

9 months ago

can we grab the project tho in the editor

?

yoelfdz

9 months ago

Once they share the package we

'll be able to open it in the editor yes

yoelfdz

9 months ago

@Andreas Møller

Could you update us on the streaming chatbot package please

?

yoelfdz

9 months ago

If only you could share how you have implemented the streaming possibility with a custom action it may be enough

. Already built a chatbot but lacking that and need to implement for the MVP

/attachments/1212561931215573082/1217795285921828906/CleanShot_2024-03-14_at_12.23.002x.png

Andreas Møller

9 months ago

async function KapaQuery(args, ctx) {
try {
const response = await fetch(
`https://api.kapa.ai/query/v1/${args.thread ? `thread/${args.thread}/`: ''}stream?query=${args.query}`,
{
method: "GET",
headers: {
"X-API-TOKEN": "<API TOKEN>",
},
}
);
if (response.status === 200) {
const reader = response.body
.pipeThrough(new TextDecoderStream())
.getReader();
const delimiter = "\u241E";
let buffer = "";
while (true) {
const result = await reader.read();
buffer += result.value;
const chunks = buffer.split(delimiter);
for (let i in chunks) {
if (i < chunks.length - 1) {
ctx.triggerActionEvent("chunk", JSON.parse(chunks[i]))
} else {
buffer = chunks[i];
}
}
if (result.done) {
ctx.triggerActionEvent("end")
break;
}
}
} else {
const message = await response.text();
ctx.triggerActionEvent("error". message)
console.error("Error fetching data:", message);
}
} catch (error) {
ctx.triggerActionEvent("error". error)
console.error("Error fetching thread data:", error);
console.error(`Thread request failed: ${error.message}`);
}

}

This is specific for calling kapa

.ai

, so it will be slightly different for OpenAI

.

🔥1

Andreas Møller

9 months ago

There is an article here specifically for openAI

https://blog.georgeck.me/how-to-handle-streaming-in-openai-gpt-chat-completions

yoelfdz

9 months ago

Awesome thanks

, will try to do it myself for flowise

. Here

's their docs on it

:

https://docs.flowiseai.com/using-flowise/streaming

I guess I can simply implement that in a custom action

, hopefully toddle manages this properly

Andreas Møller

9 months ago

Flowise uses websocketsm so that will be a slightly different setup

. But its is still possilbe with custom actions

👍1

Devin McPaul

8 months ago

To confirm

@Andreas Møller

, we can do a similar setup in Supabase for live AI streaming

, correct

?

Bishop

7 months ago

@Andreas Møller

you mentioned providing us some kind of package or code to help us do streaming

.

. is that available yet

?

Andreas Møller

7 months ago

Hey I did look into it

, but I dont think there is a way to build the package that works well for all cases

.

Andreas Møller

7 months ago

We are planning some big updates to our API tools this quater which includes support for response streaming

🙂

❤️4

@Andreas Møller

async function KapaQuery(args, ctx) {
try {
const response = await fetch(
`https://api.kapa.ai/query/v1/${args.thread ? `thread/${args.thread}/`: ''}stream?query=${args.query}`,
{
method: "GET",
headers: {
"X-API-TOKEN": "<API TOKEN>",
},
}
);
if (response.status === 200) {
const reader = response.body
.pipeThrough(new TextDecoderStream())
.getReader();
const delimiter = "\u241E";
let buffer = "";
while (true) {
const result = await reader.read();
buffer += result.value;
const chunks = buffer.split(delimiter);
for (let i in chunks) {
if (i < chunks.length - 1) {
ctx.triggerActionEvent("chunk", JSON.parse(chunks[i]))
} else {
buffer = chunks[i];
}
}
if (result.done) {
ctx.triggerActionEvent("end")
break;
}
}
} else {
const message = await response.text();
ctx.triggerActionEvent("error". message)
console.error("Error fetching data:", message);
}
} catch (error) {
ctx.triggerActionEvent("error". error)
console.error("Error fetching thread data:", error);
console.error(`Thread request failed: ${error.message}`);
}

}

This is specific for calling kapa

.ai

, so it will be slightly different for OpenAI

.

Chris Laupama

6 months ago

How are you handling the response chunks

?

It

’s the one part I

’m getting stuck on

. I

’m trying to concatenate the delta to a variable but it

’s just not working

.

Any tips here

?

@Chris Laupama

How are you handling the response chunks

’m trying to concatenate the delta to a variable but it

yoelfdz

6 months ago

I

've done it by having a token event on the streamingAPI custom action

, that token is fired every time a character is received and I simply concatenate those characters

. If variable X has value

"This is a tes

" and I receive the last character

"t

"

, use the concatenade formula adding first the Variable X value and concatenate the token event value

.

It works great for me

Chris Laupama

6 months ago

Yeah I

’ve got an action event as well where I

’m grabbing the delta and concatenating it to a variable as well

. I take the current value of the variable and then concat to it

.

I

’m wondering if I

’m doing something wrong

…

Mind running me through yours

?

yoelfdz

6 months ago

DM me

/attachments/1212561931215573082/1242219204887711936/Zight_Recording_2024-05-21_at_08.47.10_AM.mp4

Chris Laupama

6 months ago

Maybe something

@Andreas Møller

is familiar with given the implementation with

@

😯2

@Andreas Møller

We are planning some big updates to our API tools this quater which includes support for response streaming

🙂

𝓜𝓪𝔁𝓲

1 month ago

Is there anything new regarding response streaming

? Maybe also websocket support

?

@𝓜𝓪𝔁𝓲

Is there anything new regarding response streaming

? Maybe also websocket support

?

Andreas Møller

1 month ago

Can I get back to you about response streaming tomorrow

?

😉

👀2

@Andreas Møller

Can I get back to you about response streaming tomorrow

?

😉

𝓜𝓪𝔁𝓲

1 month ago

Of course

😉

Any news really appreciated

@Andreas Møller

Can I get back to you about response streaming tomorrow

?

😉

𝓜𝓪𝔁𝓲

1 month ago

Soooo

🙂

Pretty hyped haha

Andreas Møller

1 month ago

🙂

Streaming from OpenAI into a div?

Bishop

Bishop

yoelfdz

Andreas Møller

yoelfdz

yoelfdz

Bishop

Bishop

yoelfdz

Andreas Møller

yoelfdz

Bishop

yoelfdz

Andreas Møller

yoelfdz

Andreas Møller

Bishop

yoelfdz

Bishop

yoelfdz

Andreas Møller

yoelfdz

Andreas Møller

Bishop

yoelfdz

yoelfdz

yoelfdz

Andreas Møller

Andreas Møller

yoelfdz

Andreas Møller

Devin McPaul

Bishop

Andreas Møller

Andreas Møller

Chris Laupama

yoelfdz

Chris Laupama

yoelfdz

Chris Laupama

𝓜𝓪𝔁𝓲

Andreas Møller

𝓜𝓪𝔁𝓲

𝓜𝓪𝔁𝓲

Andreas Møller

Stop scrolling. Start building.

Chat with @team-toddle

Video call our CEO

Get in touch!

Product

Company

Resources

Community