fixes ray-project#59218

```python
import asyncio
from ray import serve
from typing import List

@serve.deployment(max_ongoing_requests=1000)
class MyDeployment:
    async def __call__(self) -> List[int]:
        await asyncio.sleep(.01)
        return 1

app = MyDeployment.bind()

```

testing with 100 users

Metric | With Change | Master | Δ (Master – With Change)
-- | -- | -- | --
Requests | 128,197 | 128,136 | –61
Fails | 1,273 | 1,363 | +90
Median (ms) | 170 | 170 | 0
95%ile (ms) | 260 | 260 | 0
99%ile (ms) | 310 | 320 | +10 ms
Average (ms) | 178.66 | 178.99 | +0.33 ms
Min (ms) | 9 | 8 | –1 ms
Max (ms) | 1,171 | 2,995 | +1,824 ms
Average size (bytes) | 0.99 | 0.99 | 0
Current RPS | 538.8 | 558.2 | +19.4
Current Failures/s | 0 | 0 | 0

---------

Signed-off-by: abrar <[email protected]>
Signed-off-by: kriyanshii <[email protected]>

fixes #59218

```python
import asyncio
from ray import serve
from typing import List

@serve.deployment(max_ongoing_requests=1000)
class MyDeployment:
    async def __call__(self) -> List[int]:
        await asyncio.sleep(.01)
        return 1

app = MyDeployment.bind()

```

testing with 100 users

Metric | With Change | Master | Δ (Master – With Change)
-- | -- | -- | --
Requests | 128,197 | 128,136 | –61
Fails | 1,273 | 1,363 | +90
Median (ms) | 170 | 170 | 0
95%ile (ms) | 260 | 260 | 0
99%ile (ms) | 310 | 320 | +10 ms
Average (ms) | 178.66 | 178.99 | +0.33 ms
Min (ms) | 9 | 8 | –1 ms
Max (ms) | 1,171 | 2,995 | +1,824 ms
Average size (bytes) | 0.99 | 0.99 | 0
Current RPS | 538.8 | 558.2 | +19.4
Current Failures/s | 0 | 0 | 0

---------

Signed-off-by: abrar <[email protected]>

fixes ray-project#59218

```python
import asyncio
from ray import serve
from typing import List

@serve.deployment(max_ongoing_requests=1000)
class MyDeployment:
    async def __call__(self) -> List[int]:
        await asyncio.sleep(.01)
        return 1

app = MyDeployment.bind()

```

testing with 100 users

Metric | With Change | Master | Δ (Master – With Change)
-- | -- | -- | --
Requests | 128,197 | 128,136 | –61
Fails | 1,273 | 1,363 | +90
Median (ms) | 170 | 170 | 0
95%ile (ms) | 260 | 260 | 0
99%ile (ms) | 310 | 320 | +10 ms
Average (ms) | 178.66 | 178.99 | +0.33 ms
Min (ms) | 9 | 8 | –1 ms
Max (ms) | 1,171 | 2,995 | +1,824 ms
Average size (bytes) | 0.99 | 0.99 | 0
Current RPS | 538.8 | 558.2 | +19.4
Current Failures/s | 0 | 0 | 0

---------

Signed-off-by: abrar <[email protected]>

fixes ray-project#59218

```python
import asyncio
from ray import serve
from typing import List

@serve.deployment(max_ongoing_requests=1000)
class MyDeployment:
    async def __call__(self) -> List[int]:
        await asyncio.sleep(.01)
        return 1

app = MyDeployment.bind()

```

testing with 100 users

Metric | With Change | Master | Δ (Master – With Change)
-- | -- | -- | --
Requests | 128,197 | 128,136 | –61
Fails | 1,273 | 1,363 | +90
Median (ms) | 170 | 170 | 0
95%ile (ms) | 260 | 260 | 0
99%ile (ms) | 310 | 320 | +10 ms
Average (ms) | 178.66 | 178.99 | +0.33 ms
Min (ms) | 9 | 8 | –1 ms
Max (ms) | 1,171 | 2,995 | +1,824 ms
Average size (bytes) | 0.99 | 0.99 | 0
Current RPS | 538.8 | 558.2 | +19.4
Current Failures/s | 0 | 0 | 0

---------

Signed-off-by: abrar <[email protected]>

## Why are these changes needed?

The `test_router_queue_len_metric` test was flaky because the router
queue length gauge has a 100ms throttle
(`RAY_SERVE_ROUTER_QUEUE_LEN_GAUGE_THROTTLE_S`) that can skip updates
when they happen too quickly.

When replica initialization sets the gauge to 0 and a request
immediately updates it to 1, the second update may be throttled, causing
the test to see 0 instead of 1.

## Related issue number

Fixes flaky test introduced in #59233 after #60139 added throttling.

---------

Signed-off-by: Seiji Eicher <[email protected]>

## Why are these changes needed?

The `test_router_queue_len_metric` test was flaky because the router
queue length gauge has a 100ms throttle
(`RAY_SERVE_ROUTER_QUEUE_LEN_GAUGE_THROTTLE_S`) that can skip updates
when they happen too quickly.

When replica initialization sets the gauge to 0 and a request
immediately updates it to 1, the second update may be throttled, causing
the test to see 0 instead of 1.

## Related issue number

Fixes flaky test introduced in ray-project#59233 after ray-project#60139 added throttling.

---------

Signed-off-by: Seiji Eicher <[email protected]>
Signed-off-by: jinbum-kim <[email protected]>

## Why are these changes needed?

The `test_router_queue_len_metric` test was flaky because the router
queue length gauge has a 100ms throttle
(`RAY_SERVE_ROUTER_QUEUE_LEN_GAUGE_THROTTLE_S`) that can skip updates
when they happen too quickly.

When replica initialization sets the gauge to 0 and a request
immediately updates it to 1, the second update may be throttled, causing
the test to see 0 instead of 1.

## Related issue number

Fixes flaky test introduced in ray-project#59233 after ray-project#60139 added throttling.

---------

Signed-off-by: Seiji Eicher <[email protected]>
Signed-off-by: 400Ping <[email protected]>

## Why are these changes needed?

The `test_router_queue_len_metric` test was flaky because the router
queue length gauge has a 100ms throttle
(`RAY_SERVE_ROUTER_QUEUE_LEN_GAUGE_THROTTLE_S`) that can skip updates
when they happen too quickly.

When replica initialization sets the gauge to 0 and a request
immediately updates it to 1, the second update may be throttled, causing
the test to see 0 instead of 1.

## Related issue number

Fixes flaky test introduced in ray-project#59233 after ray-project#60139 added throttling.

---------

Signed-off-by: Seiji Eicher <[email protected]>

fixes ray-project#59218

```python
import asyncio
from ray import serve
from typing import List

@serve.deployment(max_ongoing_requests=1000)
class MyDeployment:
    async def __call__(self) -> List[int]:
        await asyncio.sleep(.01)
        return 1

app = MyDeployment.bind()

```

testing with 100 users

Metric | With Change | Master | Δ (Master – With Change)
-- | -- | -- | --
Requests | 128,197 | 128,136 | –61
Fails | 1,273 | 1,363 | +90
Median (ms) | 170 | 170 | 0
95%ile (ms) | 260 | 260 | 0
99%ile (ms) | 310 | 320 | +10 ms
Average (ms) | 178.66 | 178.99 | +0.33 ms
Min (ms) | 9 | 8 | –1 ms
Max (ms) | 1,171 | 2,995 | +1,824 ms
Average size (bytes) | 0.99 | 0.99 | 0
Current RPS | 538.8 | 558.2 | +19.4
Current Failures/s | 0 | 0 | 0

---------

Signed-off-by: abrar <[email protected]>
Signed-off-by: peterxcli <[email protected]>

## Why are these changes needed?

The `test_router_queue_len_metric` test was flaky because the router
queue length gauge has a 100ms throttle
(`RAY_SERVE_ROUTER_QUEUE_LEN_GAUGE_THROTTLE_S`) that can skip updates
when they happen too quickly.

When replica initialization sets the gauge to 0 and a request
immediately updates it to 1, the second update may be throttled, causing
the test to see 0 instead of 1.

## Related issue number

Fixes flaky test introduced in ray-project#59233 after ray-project#60139 added throttling.

---------

Signed-off-by: Seiji Eicher <[email protected]>
Signed-off-by: peterxcli <[email protected]>

## Why are these changes needed?

The `test_router_queue_len_metric` test was flaky because the router
queue length gauge has a 100ms throttle
(`RAY_SERVE_ROUTER_QUEUE_LEN_GAUGE_THROTTLE_S`) that can skip updates
when they happen too quickly.

When replica initialization sets the gauge to 0 and a request
immediately updates it to 1, the second update may be throttled, causing
the test to see 0 instead of 1.

## Related issue number

Fixes flaky test introduced in ray-project#59233 after ray-project#60139 added throttling.

---------

Signed-off-by: Seiji Eicher <[email protected]>
Signed-off-by: peterxcli <[email protected]>