Question 206713 in Azure Question posted in
The official documentation to get you started can be found here.

what should I do to make the azure openai return stream events with java 11

I am using this code to return a stream response when using azure openai api:

public void getChatCompletion(SseEmitter emitter, String prompt, String azureOpenaiKey) {
        String endpoint = "https://xxxx.openai.azure.com/";
        String deploymentOrModelId = "xxx-ai";
        OpenAIClient client = new OpenAIClientBuilder()
                .endpoint(endpoint)
                .credential(new AzureKeyCredential(azureOpenaiKey))
                .buildClient();
        List<ChatMessage> chatMessages = new ArrayList<>();
        chatMessages.add(new ChatMessage(ChatRole.SYSTEM).setContent("You are a helpful assistant."));
        chatMessages.add(new ChatMessage(ChatRole.USER).setContent(prompt));
        ChatCompletionsOptions options = new ChatCompletionsOptions(chatMessages);
        options.setStream(true);
        options.setModel("gpt-3.5-turbo-0613");
        IterableStream<ChatCompletions> chatCompletions = client.getChatCompletionsStream(deploymentOrModelId, options);
        chatCompletions.forEach(completions->{
            try {
                emitter.send(completions);
            } catch (Exception e) {
                log.error("send emit message error", e);
            }
        });
    }

this code works but I found it is not a stream result, it still look like a http reqeust, send a request and return the full response onece, is it possible to return the response words by words? I have already read the official demo: https://learn.microsoft.com/en-us/java/api/overview/azure/ai-openai-readme?view=azure-java-preview#chat-completions. Am I missing something? This is my java dependencies using gradle:

implementation'com.azure:azure-ai-openai:1.0.0-beta.2'
Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    Finally I found a better way to return stream response, change the client to async to get fast response:

    public void getAsyncChatCompletion(SseEmitter emitter, String prompt, String azureOpenaiKey) {
        String endpoint = "https://xxx.openai.azure.com/";
        String deploymentOrModelId = "xxx-ai";
        OpenAIAsyncClient client = new OpenAIClientBuilder()
                .endpoint(endpoint)
                .credential(new AzureKeyCredential(azureOpenaiKey))
                .buildAsyncClient();
        List<ChatMessage> chatMessages = new ArrayList<>();
        chatMessages.add(new ChatMessage(ChatRole.SYSTEM).setContent("You are a helpful assistant."));
        chatMessages.add(new ChatMessage(ChatRole.USER).setContent(prompt));
        ChatCompletionsOptions options = new ChatCompletionsOptions(chatMessages);
        options.setStream(true);
        options.setModel("gpt-3.5-turbo-0613");
        Flux<ChatCompletions> chatCompletions = client.getChatCompletionsStream(deploymentOrModelId, options);
        chatCompletions.subscribe(chatCompletion -> {
                    try {
                        SseChatService.getInstance().appendMsg(chatCompletion, RequestContext.getRequestId());
                        emitter.send(chatCompletion);
                    } catch (Exception e) {
                        log.error("send emit message error", e);
                    }
                },
                throwable -> {
                    log.error("flux response facing error:", throwable.getMessage());
                },
                () -> {
                    emitter.complete();
                });
    }

  2. 0

    Many thanks to Dolphin.
    When I call Azure OpenAI Java API refer to azure openAI offical doc, the response is:"resouces not found (404)", I am really going crazy.
    After adding something like "options.setModel("gpt-3.5-turbo-0613")" in my code, it finally works.
    If I didn’t see "options.setModel()" in your post, it might take a few days to find the real answer.
    I did not find any ".setModel()" code snippet in the Azure openAI Java docs or other azure offical demo.Just for my curiosity,where did you find this code?

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search