Streaming

Streaming allows responses to be delivered progressively in real-time, providing a continuous flow of information as it’s generated. This feature enables an experience similar to ChatGPT, where responses are displayed word-by-word or in chunks, enhancing interactivity in applications.

When streaming: true is set, the API returns data in a streamed format instead of waiting for the entire response to be generated. This requires handling the streamed data as it arrives, using a ReadableStream.

Request with Streaming

To enable streaming, include "streaming": true in the request body. This setup initiates a real-time response where data is sent incrementally to the client.

Sample Request for Streaming

const serviceUrl = "https://78b8-46-188-242-94.ngrok-free.app/conversation"
const apiKey = "YOUR_RAGAPI_API_KEY"
const pineconeIndexName = "YOUR_PINECONE_INDEX"
const pineconeNamespace = "259520d2-d7e2-438d-8d4e-0eb8f38d5273"
const query = "Where was Ignatius the blockchain coming from?"

const response = await fetch(serviceUrl, {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    "x-api-key": apiKey,
  },
  body: JSON.stringify({
    pineconeIndexName,
    pineconeNamespace,
    query,
    streaming: true,
  }),
})

const reader = response.body.getReader()
const decoder = new TextDecoder("utf-8")

while (true) {
  const { done, value } = await reader.read()
  if (done) break
  const text = decoder.decode(value, { stream: true })
  // Split by double newlines to get individual SSE messages
  const messages = text.split("\n\n")

  for (const message of messages) {
    // Check if it's a data message
    if (message.startsWith("data: ")) {
      try {
        // Remove the 'data: ' prefix and parse the JSON
        const jsonStr = message.slice(6)
        const parsedObject = JSON.parse(jsonStr)
        console.log(parsedObject.content)
      } catch (error) {
        console.error("Failed to parse SSE message:", message, error)
      }
    }
  }
}

How to Use Streaming

1. Read Streamed Data: Use the ReadableStream's getReader() to access the response data in chunks.
2. Decode Each Chunk: Convert each chunk from a byte stream to readable text using TextDecoder.
3. Process JSON Data: Split the result by lines, filter empty lines, and parse each JSON object. The parsed object’s content can be displayed progressively.

Advantages of Streaming

• Interactivity: Provides immediate feedback by displaying partial answers.
• Reduced Latency: Users don’t wait for the entire response, which improves the perception of speed.
• Enhanced User Experience: Ideal for applications requiring conversational interactions, real-time data display, or large responses.