Bedrock Runtime converseStream doesn't seem to be streaming

[gazzoy]

Describe the bug

When calling the converseStream() method from the AWS PHP SDK, the result is supposed to be returned as soon as possible, then we can iterate stream value in the result until the streaming finished.

However, debug logs show that when calling converseStream() method, we need to wait for the whole streaming finished, then it returns results, which seems weird to me.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

converseStream() should return the result as soon as possible.

Current Behavior

converseStream() does NOT return the result as soon as possible rather it waits until the streaming finished, then it returns the result.

Reproduction Steps

Belows are example code to describe the issue.

We see debug logs like:

2024-10-31 13:10:00 calling converseStream()
2024-10-31 13:10:30 called converseStream()  <-- it waits 30 seconds until the streaming finished, which is weird.
2024-10-31 13:10:30 looping stream()
2024-10-31 13:10:30 looped stream() <- it doesn't take time to loop as its streaming seems already done.

I might be wrong, but should converseStream() work like following?

2024-10-31 13:10:00 calling converseStream()
2024-10-31 13:10:00 called converseStream()  <-- it returns the result as soon as possible
2024-10-31 13:10:00 looping stream()
2024-10-31 13:10:30 looped stream() <-- it takes 30 seconds to finish the streaming.

<?php

use Aws\BedrockRuntime\BedrockRuntimeClient;
use Aws\Credentials\CredentialProvider;

$bedrockClient = new BedrockRuntimeClient([
    'version' => 'latest',
    'region' => 'ap-northeast-1',
    'credentials' => CredentialProvider::env(),
]);

$input = 'Describe about aws in 5k texts';

\Log::debug('calling converseStream()');
$result = $bedrockClient->converseStream([
    'modelId' => 'anthropic.claude-3-5-sonnet-20240620-v1:0', // note: Claude 3.5 Sonnet
    'messages' => [
        [
            'content' => [
                [
                    'text' => "\n\nHuman:{$input} \n\nAssistant:",
                ],
            ],
            'role' => 'user',
        ],
    ],
]);
\Log::debug('called converseStream()');

\Log::debug('looping stream()');
foreach ($result['stream'] as $eachStreamChunk)
{
    // do something
}
\Log::debug('looped stream()');

Possible Solution

No response

Additional Information/Context

No response

SDK version used

3.324.11

Environment details (Version of PHP (php -v)? OS name and version, etc.)

% php -v                                                                                                                                                 
PHP 8.3.12 (cli) (built: Sep 27 2024 03:53:05) (NTS) 
Copyright (c) The PHP Group
 Zend Engine v4.3.12, Copyright (c) Zend Technologies
     with Zend OPcache v8.3.12, Copyright (c), by Zend Technologies
     with Xdebug v3.3.2, Copyright (c) 2002-2024, by Derick Rethans

[gazzoy]

#2817 (comment) could be related to this.

[HagnMarkus]

Same here with invokeModelWithResponseStreamAsync

[yenfryherrerafeliz]

Hi @gazzoy, thanks for reporting this issue. I will add an item into our backlog for addressing this as soon as we can, but for now would you please be able to use the following workaround:

$result = $bedrockClient->converseStream([
    'modelId' => 'anthropic.claude-3-5-sonnet-20240620-v1:0', // note: Claude 3.5 Sonnet
    'messages' => [
        [
            'content' => [
                [
                    'text' => "\n\nHuman:{$input} \n\nAssistant:",
                ],
            ],
            'role' => 'user',
        ],
    ],
    '@http' => [
          'stream' => true
    ]
]);

We are just adding a flag to specify that the response should be streamed.

Please let me know if that helps you.

Thanks!

[gazzoy]

@yenfryherrerafeliz Your workaround works fine! Thanks for supporting this.