Release notes from ray

Ray-2.54.0

2026-02-19T00:39:47Z

Ray Data

🎉 New Features

Add checkpointing support to Ray Data (#59409)
Compute Expressions: list operations (#59346), fixed-size arrays (#58741), string padding (#59552), logarithmic (#59549), trigonometric (#59712), arithmetic (#59678), and rounding (#59295)
Add sql_params support to read_sql (#60030)
Add AsList aggregation (#59920)
Support CountDistinct aggregate (#59030)
Add credential provider abstraction for Databricks UC datasource (#60457)
Support callable classes for UDFExpr (#56725)
Add autoscaler metrics to Data Dashboard (#60472)
Add optional filesystem parameter to download expression (#60677)
Allow specifying partitioning style or flavor in write_parquet() (#59102)
New cluster autoscaler enabled by default (#60474)

💫 Enhancements

Improve numerical stability in scalers by handling near-zero values (#60488)
Export dataset operator output schema to event logger (#60086)
Iceberg: add retry policy for Storage + Catalog writes (#60620)
Iceberg: remove calls to Catalog Table in write tasks (#60476)
Expose logical operators and rules via package exports (#60297, #60296)
Demote Sort from requiring preserve_order (#60555)
Improve appearance of repr(dataset) (#59631)
Allow configuring DefaultClusterAutoscalerV2 thresholds via env vars (#60133)
Use Arrow IPC for Arrow Schema serialization/deserialization (#60195)
Store _source_paths in object store to prevent excessive spilling during read task serialization (#59999)
Add more shuffle fusion rules (#59985)
Enable and tune DownstreamCapacityBackpressurePolicy (#59753)
Enable concurrency cap backpressure with tuning (#59392)
Set default actor pool scale up threshold to 1.75 (#59512)
Don't downscale actors if the operator hasn't received any inputs (#59883)
Don't reserve GPU budget for non-GPU tasks (#59789)
Only return selected data columns in hive-partitioned Parquet files (#60236)
Ordered + FIFO bundle queue (#60228)
Add node_id, pid, attempt number for hanging tasks (#59793)
Revise resource allocator task scheduling to factor in pending task outputs (#60639)
Track block serialization time (#60574)
Use metrics from OpRuntimeMetrics for progress (#60304)
Tabular form for streaming executor op metrics (#59774)
Info-log cluster scale-up decisions (#60357)
Use plain mode instead of grid mode for OpMetrics logging (#59907)
Progress reporting refactors (#59350, #59629, #59880)
Remove deprecated TENSOR_COLUMN_NAME constant (#60573)
Remove meta_provider parameter (#60379)
Decouple Ray Train from Ray Data by removing top-level ray.data imports (#60292)
Move extension types to ray.data (#59420)
Skip upscaling validation warning for fixed-size actor pools (#60569)
Make StatefulShuffleAggregation.finalize allow incremental streaming (#59972)
Revisit OutputSplitter semantics to avoid unnecessary buffer accumulation (#60237)
Update to PyArrow 23 (#60739, #59489)
Add BackpressurePolicy to streaming executor progress bar (#59637)
Support Arrow-based transformations for preprocessors (#59810)
StandardScaler preprocessor with Arrow format (#59906)
OneHotEncoder with Arrow format (#59890)

🔨 Fixes

Fuse MapBatches even if they modify the row count (#60756)
Don't push limit past map_batches by default (#60448)
Fix wrong type hint of other dataset in zip and union (#60653)
Fix ActorPoolMapOperator to guarantee dispatch of all given inputs (#60763)
Fix ArrowInvalid error when backfilling missing fields from map tasks (#60643)
Fix attribute error in UnionOperator.clear_internal_output_queue (#60538)
Fix DefaultClusterAutoscalerV2 raising KeyError: 'CPU' (#60208)
Fix ReorderingBundleQueue handling of empty output sequences (#60470)
Fix task completion time without backpressure grafana panel metric name (#60481)
Fix Union operator blocking when preserve_order is set (#59922)
Fix autoscaler requesting empty resources instead of previous allocation when not scaling up (#60321)
Fix autoscaler not respecting user-configured resource limits (#60283)
Fix DefaultAutoscalerV2 not scaling nodes from zero (#59896)
Fix Iceberg warning message (#60044)
Fix Parquet datasource path column support (#60046)
Fix ProgressBar with use_ray_tqdm (#59996)
Fix stale stats on refit for preprocessors (#60031)
Fix StreamingRepartition hang with empty upstream results (#59848)
Fix operator fusion bug to preserve UDF modifying row count (#59513)
Fix AutoscalingCoordinator double-allocating resources for multiple datasets (#59740)
Fix DownstreamCapacityBackpressurePolicy issues (#59990)
Fix AutoscalingCoordinator crash when requesting 0 GPUs on CPU-only cluster (#59514)
Fix TensorArray to Arrow tensor conversion (#59449)
Fix resource allocator not respecting max resource requirement (#59412)
Fix GPU autoscaling when max_actors is set (#59632)
Fix checkpoint filter PyArrow zero-copy conversion error (#59839)
Restore class aliases to fix deserialization of existing datasets (#59828, #59818)
Fix DataContext deserialization issue with StatsActor (#59471)

📖 Documentation

Sort references in "Loading data and Saving data" pages (#60084)
Fix inconsistent heading levels in "How to write tests" guide (#60706)
Clarify resource_limits refers to logical resources (#60109)
Update read_lance doc (#59673)
Fix broken link in read_unity_catalog docstring (#59745)
Fix bug in docs for enable_true_multi_threading (#60515)
Add more education around transformations (#59415)

Ray Serve

🎉 New Features

Queue-based autoscaling for TaskConsumer deployments (phase 1). Introduces a QueueMonitor actor that queries message brokers (Redis, RabbitMQ) for queue length, enabling TaskConsumer scaling based on pending tasks rather than HTTP load. (#59430)
Default autoscaling parameters for custom policies. New apply_autoscaling_config decorator allows custom autoscaling policies to automatically benefit from Ray Serve's standard parameters (delays, scaling factors, bounds) without reimplementation. (#58857)
label_selector and bundle_label_selector in Serve deployments. Deployments can now specify node label selectors for scheduling and bundle-level label selectors for placement groups, useful for targeting specific hardware (e.g., TPU topologies). (#57694)
Deployment-level autoscaling observability. The controller now emits a structured JSON serve_autoscaling_snapshot log per autoscaling-enabled deployment each control-loop tick, with an event summarizer that reduces duplicate logs. (#56225)
Batching with multiplexing support. Batching now guarantees each batch contains requests for the same multiplexed model, enabling correct multiplexed model serving with @serve.batch. (#59334)

💫 Enhancements

Replica routing data structure optimizations. O(1) pending-request lookups, cached replica lists, lazy cleanup, optimized retry insertion, and metrics throttling yield significant routing performance improvements. (#60139)
New operational metrics suite. Added long-poll metrics, replica lifecycle metrics, app/deployment status metrics, proxy health and request routing delay metrics, event loop utilization metrics, and controller health metrics — greatly improving monitoring and debugging capabilities. (#59246, #59235, #59244, #59238, #59535, #60473)
Autoscaling config validation. lookback_period_s must now be greater than metrics_interval_s, preventing silent misconfigurations. (#59456)
Cross-version root_path support for uvicorn. root_path now works correctly across all uvicorn versions, including >=0.26.0 which changed how root_path is processed. (#57555)
Preserve user-set gRPC status codes. When deployments raise exceptions after setting a gRPC status code on the context, that code is now correctly propagated to the client instead of being overwritten with INTERNAL. Error messages are truncated to 4 KB to respect HTTP/2 trailer limits. (#60482)
Replica ThreadPoolExecutor capped to num_cpus. The user-code event loop's default ThreadPoolExecutor is now limited to the deployment's num_cpus, preventing oversubscription when using asyncio.to_thread. (#60271)
Generic actor registration API for shutdown cleanup. Deployments can register auxiliary actors (e.g., PrefixTreeActor) with the controller for automatic cleanup on serve.shutdown(), eliminating cross-library import dependencies. (#60067)
Deployment config logging in controller. Deployment configurations are now logged in the controller for easier debugging and auditability. (#59222, #59501)
Pydantic v1 deprecation warning. A FutureWarning is now emitted at ray.init() when Pydantic v1 is detected, as support will be removed in Ray 2.56. (#59703)

🔨 Fixes

Fixed tracing signature mismatch across processes. Resolved TypeError: got an unexpected keyword argument _ray_trace_ctx when calling actors from a different process than the one that created them (e.g., serve start + dashboard interaction). (#59634)
Fixed ingress deployment name collision. Ingress deployment name was incorrectly modified when a child deployment shared the same name, causing routing failures. (#59577)
Fixed downstream deployment over-provisioning. Downstream deployments no longer over-provision replicas when receiving DeploymentResponse objects. (#60747)
Fixed replicas hanging forever during draining. Replicas no longer hang indefinitely when requests are stuck during the draining phase. (#60788)
Fixed TaskProcessorAdapter shutdown during rolling updates. Removed shutdown() from __del__, which was broadcasting a kill signal to all Celery workers instead of just the local one, breaking rolling updates. (#59713)
Fixed Windows test failures. Resolved tracing file handle cleanup on Windows, skipped incompatible gRPC and tracing tests on Windows. (#60078, #60356, #60393, #59771)
Fixed flaky tests. Addressed gauge throttling race in test_router_queue_len_metric, ensured proxy replica queue cache is populated before GCS failure tests, and added metrics server readiness checks. (#60333, #60466, #60468)
Fixed distilbert test segfault. Worked around a pyarrow/jemalloc crash triggered by specific import ordering of FastAPI, torch, and TensorFlow. (#60478)

📖 Documentation

Improved autoscaling documentation. Clarified the relationship between delays, metric push intervals, and the autoscaling control loop. (#59475)
New example: video analysis inference. End-to-end notebook demonstrating a Serve application for scene change detection, - tagging, and video description. (#59859)
New examples: model multiplexing and model composition. Published workload-based examples for forecasting with model multiplexing and recommendation systems with model composition. (#59166)
Model registry integration guide. Added documentation for integrating Serve with model registries (e.g., MLflow). (#59080)
Fixed broken documentation links. Resolved 404 errors for async inference, MLflow registry example, and LLM code examples. (#59917, #60071, #59520, #59521, #60181)
Fixed monitoring docs. Corrected target replicas metric emission to enable time-series comparison with actual replicas. (#59571)
Async inference template. Added an end-to-end template for building asynchronous inference applications with Ray Serve. (#58393, #59926)

🏗 Architecture refactoring

Environment variable cleanup (5-part series). Removed deprecated and redundant env vars (RAY_SERVE_DEFAULT_HTTP_HOST, RAY_SERVE_DEFAULT_HTTP_PORT, RAY_SERVE_DEFAULT_GRPC_PORT, RAY_SERVE_HTTP_KEEP_ALIVE_TIMEOUT_S, RAY_SERVE_REQUEST_PROCESSING_TIMEOUT_S, RAY_SERVE_ENABLE_JSON_LOGGING, RAY_SERVE_ALWAYS_RUN_PROXY_ON_HEAD_NODE), cleaned up legacy constant fallbacks, and added documentation for previously undocumented env vars (e.g., RAY_SERVE_CONTROLLER_MAX_CONCURRENCY, RAY_SERVE_ROOT_URL, proxy health check settings, and fault tolerance params). Users relying on removed env vars should migrate to the Serve config API (http_options, grpc_options, LoggingConfig). (#59470, #59619, #59647, #59963, #60093)

Ray Train

🎉 New Features

Add TPU multi-slice support to JaxTrainer (#58629)
Update async validation API (#59428)
Add a CallbackManager and guardrail some callback hooks (#60117)
Add inter-execution file shuffling for deterministic multi-epoch training (#59528)
Resume validations on driver restoration (#59270)

💫 Enhancements

Pass ray remote args to validation task (#60203)
Deprecate Predictor API (#60305)
Increase worker group start default timeout to 60s (#60376)
Unify PlacementGroup and SlicePlacementGroup interface in WorkerGroup (#60116)
Cleanup zombie RayTrainWorker actors (#59872)
Add usage telemetry for checkpointing and validation (#59490)
Validate that validation is called with a checkpoint (#60548)
Replace pg.ready() with pg.wait() in worker group (#60568)
Rename DatasetsSetupCallback to DatasetsCallback (#59423)
Update "Checkpoint Report Time" metric title to "Cumulative Checkpoint Report Time" (#58470)
Add training failed error back to failure policy log (#59957)
Decouple Ray Train from Ray Data by removing top-level imports (#60292)

🔨 Fixes

Add try-except for pg.wait() (#60743)
TrainController reraises AsyncioActorExit (#59461)

📖 Documentation

Add a JaxTrainer template (#59842)
Update Jax doc to include GPU and multi-slice TPU support (#60593)
Document checkpoint_upload_fn backend and cuda:nccl backend support (#60541)
Rename checkpoint_upload_func to checkpoint_upload_fn in docs (#60390)
Fix Ray Train workloads and PyTorch with ASHA templates (#60537)
Publish Ray Train workload example (#58936)

Ray Tune

🔨 Fixes

Avoid file deletion race by using unique tmp file names (#60556)

Ray LLM

🎉 New Features

Add /tokenize and /detokenize endpoints (#59787)
Add /collective_rpc endpoint for RLHF weight synchronization (#59529)
Add Control Plane API for Sleep/Wakeup (#59455)
Add Pause/Resume Control Plane API (#59523)
Add support for classification and scoring models (#59499)
Add pooling parameter (#59534)
Support vLLM structured outputs with backward-compat for guided_decoding (#59421)
Add CPU support to Ray Serve LLM (#58334)
Add should_continue_on_error support for ServeDeploymentStage (#59395)
Support configuring HttpRequestUDF resources (#60313)

💫 Enhancements

Upgrade vLLM to 0.15.0 (#60679)
Unify schema of success and failure rows (#60572)
Prefer uniproc executor over mp executor when world_size==1 (#60403)
Use compute instead of concurrency to specify ActorPool size (#59645)
Remove DataContext overrides in Ray Data LLM Processor (#60142)
Use numpy arrays for embeddings to avoid torch.Tensor serialization overhead (#59919)
Make PrefixCacheAwareRouter imbalance threshold less surprising (#59390)
Allow tokenized_prompt without prompt in vLLMEngineStage (#59801)
Avoid passing enums through fn_constructor_kwargs (#59806)
Refactor Control Plane endpoints into mixins (#59502)
Remove CUDA_VISIBLE_DEVICES deletion workaround (#60502)

🔨 Fixes

Fix nested dict to Namespace conversion in vLLM engine initialization (#60380)
Fix JSON non-serializable ndarray exception in http_request_stage (#60299)
Exit actor on EngineDeadError to enable recovery (#60145)
Fix NIXL port conflict in prefill-decode disaggregation test (#60057)

📖 Documentation

Batch inference docs reorg and update to reflect per-stage config refactor (#59214)
Add resiliency section and refine doc code (#60594)
Add video/audio examples for vLLMEngineProcessor (#59446)
Add SGLang integration example (#58366)
Remove inaccurate statement in docs (#60425)

Ray RLlib

🎉 New Features

Add TQC (Truncated Quantile Critics) algorithm implementation (#59808)
Add LR scheduling ability to BC and MARWIL (#59067)
RLlib and Ray Tune: Hyperparameter Optimisation example (#60182)

💫 Enhancements

🔥 APPO improvements: learner pipeline performance improvements (#59544)
Improve stateful model training on offline data (#59345)
Create resource bundle per learner (#59620)
Improve env runner sampling by replacing recursive solution with iterative solution (#56082)
Improve IMPALA examples and premerge (#59927)
Remove MLAgents dependency (#59524)
Upgrade to gymnasium v1.2.2 (#59530)
Decrease log quantity for learning tests (#59005)
Update learner state warnings to the debug level (#60178)
Don't log np.nanmean warnings in EMA stats (#60408)

🔨 Fixes

Fix DQN RLModule forward methods to handle dict spaces (#60451)
Fix LearnerGroup.load_module_state() and mark as deprecated (#60354)
Fix static dimension issue in ONNX export of Torch attention models (#60102)
Fix Multi-Agent Episode concatenation for sequential environments (#59895)
Fix module episode returns metrics accumulation for shared module IDs (#60234)
Fix rollout fragment length calculation in AlgorithmConfig (#59438)
Fix checkpointable issues with cloud storages (#60440)
Update flatten_observations.py for nested spaces for ignored multi-agent (#59928)

Ray Core

🎉 New Features

Resource Isolation: unify config construction, add public docs, and expose cgroup_path in ray.init() (#59372, #60183, #60726)
Support tensor-level deduplication for NIXL (#60509)
Add CUDA IPC transport for RDT (#59838)
Register custom transport at runtime for RDT (#59255)
Support TPU v7x accelerator type for device discovery (#60338)
Introduce local port service discovery (#59613)
Cancel sync actor by checking is_canceled() (#58914)
Support labels for ray job submit --entrypoint-resource (#59735)
Add --ip option in ray attach (#59931)
Add bearer token support for remote URI downloads (#60050)
Support HTTP redirection download (#59384)
Add ray kill-actor --name/--namespace for force/graceful shutdown (#60258)

💫 Enhancements

Bound object spilling file size to avoid disk increase pressure (#60098)
Replace SHA-1 with SHA-256 for internal hash operations (#60242)
Use whitelist approach to block mutation requests from browser (#60526)
Pass authentication headers to WebSocket connections in tail_job_logs (#60346)
Add auth to Dashboard HTTP agent and client (#59891)
Use dedicated service account path for Ray auth tokens (#60409)
Update Kubernetes token auth verb to ray:write (#60411)
Replace RAY_AUTH_MODE=k8s with separate config for Kubernetes token auth (#59621)
Optimize token auth: use shared_ptr caching and avoid per-RPC construction (#59500)
Optimize OpenTelemetry metric recording calls (#59337)
Throttle infeasible resource warning (#59790)
Add default excludes for working_dir uploads (#59566)
Tell users why objects cannot be reconstructed (#59625)
Extend instance allocation timeout in autoscaler v2 (#60392)
Remove GCS centralized scheduling (#59979, #60121, #60188)
Demote stale sync message drop log to DEBUG in RaySyncer (#59616)
Migrate remaining std::unordered_map to absl::flat_hash_map (#59921)
Add missing fields to NodeDefinitionEvent proto (#60314)
Add actor and task event missing fields (#60287)
Add node id to the base event (#59242)
Add repr_name to actor_lifecycle_event (#59925)
Support ALL in exposable event config (#59878)
Support publishing events from aggregator to GCS (#55781)
Update the attempt number of actor creation task when actor restarts (#58877)
Unify node feasibility and availability checking for GPU fractions (#59278)
Update TPU utils for multi-slice compatibility (#59136)
Improve SubprocessModuleHandle.destroy_module() resource cleanup (#60172)
Support viewing PIDs for Dashboard and Runtime Env Agent (#58701)
Optimize autoscaler monitor by moving resource demand parsing outside loop (#59190)
Avoid GCS query for is_head in dashboard agent startup (#59378)
Skip reporter and event aggregator client creation in minimal mode (#59846)
Support out-of-order actors by extracting metadata when creating (RDT) (#59610)
Synchronize CUDA stream before registering for NIXL (#60072)
Atomically send/recv for two-sided ordering (RDT) (#60202)
Add get_session_name() to RuntimeContext (#59469)
Make MAX_APPLICATION_ERROR_LEN configurable via env var (#59543)
Preserve function signatures through Ray decorators (#60479)

🔨 Fixes

Fix idle_time_ms resetting for nodes not running tasks (#60581)
Fix task event loss during shutdown (#60247)
Filter bad subscriber messages from taking down GCS publisher (#60252)
Fix RAY_EXPERIMENTAL_NOSET_* environment variable parsing in accelerator managers (#60577)
Fix ray start --no-redirect-output crash (#60394)
Fix drain state propagation race condition (#59536)
Fix use-after-free race condition in OpenTelemetry gauge metric callback during shutdown (#60048)
Fix PSUTIL_PROCESS_ATTRS returning empty list on Windows (#60173)
Fix deadlock in garbage collection when holding lock (#60014)
Fix incorrect error handling in autoscaler for available_node_types on on-prem clusters (#60184)
Fix invalid status transitions in autoscaler v2 (#60412, #59550)
Fix GCS crash from race condition in MetricsAgentClient exporter initialization (#59611)
Fix tracing signature mismatch when calling actors from different processes (#59634)
Fix crash when killing actor handle from previous session (#59425)
Fix multiple deployment same name resolve (#59577)
Handle dual task errors with read-only args (#59507)
Handle exceptions raised by internal_ip() within StandardAutoscaler (#57279)
Fix uv_runtime_env_hook.py to pin worker Python version (#59768)
Fix STRICT_PACK placement groups ignoring bundle label selectors (#60170)
Fix logging bug when log value is an empty string (#59434)
Fix aggregator-to-GCS event conversion (#59783)
Raise error on tail log job error in newer Ray versions (#59506)
Fix num retries left message (#59829)
Fix psutil internal API usage in dashboard disk usage reporting (#59659)
Fix event exporter init ray check (#60073)
Prevent use-after-free error in core worker shutdown (#58435)
Fix task name inconsistency in RUNNING vs FINISHED metrics (#59893)
Fix symmetric_run using wrong condition to check GCS readiness (#59794)
Preserve Pydantic details when serialization fails (#59401)
Retry GCP project metadata updates on HTTP 412 errors (#60429)
Fix v1 autoscaler TypeError when using bundle_label_selectors (#59850)
Shorten SHA-256 hex with base32 to comply with GCP label limits (#60722)

📖 Documentation

Add initial user guide for Ray resource isolation with writable cgroups (#59051)
Add token authentication internals documentation (#59299)
Update metric exporter docs (#59874)
Add internal documentation for Port Service Discovery (#59844)
Update misleading Ray job diagram (#59940)
Add debugging logs related to pinned argument size limit (#60175)
Add slow startup tip to podman troubleshooting docs (#59942)
Clarify ray.shutdown() behavior for local vs remote clusters (#59845)
Improve placement group fault tolerance doc (#59830)
Add head-node memory growth and OOM guidance (#58695)
Add documentation for RAY_RUNTIME_ENV_BEARER_TOKEN env var (#60136)

Dashboard

💫 Enhancements

Support more panels in dashboard (#60018)
Add autoscaler metrics to Data Dashboard (#60472)
Support viewing PIDs for Dashboard and Runtime Env Agent (#58701)

🔨 Fixes

Update total for dark mode color (#60106)

Ray Wheels and Images

Add experimental Python 3.13 images (#59811, #60549)
Drop Python 3.9 support completely (#59824, #59825, #59769); wheel Python version now declared as 3.10+ (#59835)
ray-cpp wheels are now py3-none, without specific Python versions. (#59969)

Documentation

Add committership documentation (#60069)
Update contribution guide with common labels (#59473)
Add KubeRay & Volcano integration docs update (#59636)
Add RayJob InTreeAutoscaling with Kueue docs after Kueue 0.16.0 release (#59648)
Refactor LLM batch inference template (#59897)
Add async inference template (#58393)
Add RunLLM chat widget for Ray docs (#59126)
Fix various typos and broken links (#60249, #59901, #60181)
Replace Ray Tune + Train example with vanilla Ray Tune in homepage (#60229)
Add Ray technical charter (#60068)

Thanks

Thank you to everyone who contributed to this release!
@KaisennHu, @MiXaiLL76, @slfan1989, @krisselberg, @JasonLi1909, @Priya-753, @pseudo-rnd-thoughts, @zzchun, @ZacAttack, @pushpavanthar, @jjyao, @ryanaoleary, @pcmoritz, @akshay-anyscale, @HassamSheikh, @yurekami, @Hyunoh-Yeo, @ruoliu2, @nrghosh, @wxwmd, @myandpr, @J-Meyers, @trilamsr, @kouroshHakha, @limarkdcunha, @manhld0206, @jreiml, @preneond, @yuchen-ecnu, @Yicheng-Lu-llll, @AchimGaedkeLynker, @vaishdho1, @israbbani, @OneSizeFitsQuorum, @Sathyanarayanaa-T, @nadongjun, @xinyuangui2, @Rob12312368, @as-jding, @lee1258561, @popojk, @coqian, @rajeshg007, @jeffreywang-anyscale, @kamil-kaczmarek, @alexeykudinkin, @Aydin-ab, @mgchoi239, @dragongu, @edoakes, @smortime, @tk42, @abrarsheikh, @jakubzimny, @Future-Outlier, @axreldable, @owenowenisme, @g199209, @cem-anyscale, @dayshah, @akelloway, @daiping8, @dlwh, @robertnishihara, @400Ping, @matthewdeng, @antoine-galataud, @cristianjd, @Partth101, @goutamvenkat-anyscale, @codope, @seanlaii, @andrew-anyscale, @andrewsykim, @liulehui, @simonsays1980, @Sparks0219, @yifanmai, @landscapepainter, @win5923, @kangwangamd, @srinarayan-srikanthan, @KeeProMise, @srinathk10, @my-vegetable-has-exploded, @MengjinYan, @yancanmao, @yuhuan130, @ArturNiederfahrenhorst, @akyang-anyscale, @rushikeshadhav, @kongjy, @harshit-anyscale, @justinvyu, @dancingactor, @Vito-Yang, @cr7258, @marwan116, @muyihao, @DeborahOlaboye, @bveeramani, @kriyanshii, @khluu, @machichima, @Kunchd, @jonded94, @iamjustinhsu, @sampan-s-nayak, @wingkitlee0, @sunsetxh, @dkhachyan, @can-anyscale, @TimothySeah, @raulchen, @elliot-barn, @ryankert01, @xyuzh, @stephanie-wang, @hao-aaron, @simeetnayan81, @cszhu, @richardliaw, @yuanjiewei, @kyuds, @eicherseiji, @RedGrey1993, @rueian, @jeffreyjeffreywang, @crypdick, @ankur-anyscale, @aslonnie

Ray-2.53.0

2025-12-20T15:56:01Z

Highlights

Ray plans to drop support for Pydantic V1 starting version 2.56.0. Please see this RFC for details.
Ray Data now has support for bounded reading from Kafka and improved Iceberg support.

Ray Data

🎉 New Features

Autoscaling: New utilization-based cluster autoscaler for Ray Data workloads (#59353, #59362, #59366). To use this new autoscaler set RAY_DATA_CLUSTER_AUTOSCALER=V2.
Kafka Datasource: Add Kafka as a native datasource for data ingestion (#58592)
Dataset summary API: Add Dataset.summary() API for quick dataset inspection (#58862)
Iceberg support: Add Iceberg schema evolution, upsert, and overwrite support (#59210, #59335)
Graceful error handling: Add should_continue_on_error for graceful error handling in batch inference (#59212)
Datetime compute expressions: Add datetime compute expressions support (#58740)
Grouped with_column expressions: Enable expressions for grouped with_column in Ray Data (#58231)
Parallelized collation: Parallelize DefaultCollateFn, arrow_batch_to_tensors (#58821)

💫 Enhancements

Optimized Autoscaler Step Size: Optimize autoscaler to support configurable step size for actor pool scaling (#58726)
Improved Streaming Repartition: Improve streaming repartition performance (#58728)
Actor init retry: Add actor retry if there's a failure in __init__ (#59105)
Fused Repartition + MapBatches: Fuse StreamingRepartition with MapBatches operators to scale collate (#59108)
Combined repartitions: Combine consecutive repartitions for efficiency (#59145)
Prefetch buffering: Handle prefetch buffering in iter_batches (#58657)
HashShuffle block breakdown: HashShuffleAggregator breaks down blocks on finalize (#58603)
Backpressure tuning: Tune concurrency cap backpressure object store budget ratio (#58813)
Non-string ApproximateTopK: Support non-string items for ApproximateTopK aggregator (#58659)
Lance version support: Add version support to read_lance() (#58895)
Dashboard metrics: Add time_to_first_batch and get_ref_bundles metrics to data dashboard (#58912)
Iter prefetched bytes stats: Add iter_prefetched_bytes statistics tracking (#58900)
Configurable batching for iter_batches: Add configurable batching for resolve_block_refs to speed up iter_batches (#58467)
Improved dashboard metrics: Improve Ray Data dashboard metrics display (#58667)
Histogram percentiles: Update Ray Data histograms to show percentiles in data dashboard (#58650)
Deprecated API removal: Remove deprecated read_parquet_bulk API (#58970)
Block shaping option: Add disable block shaping option to BlockOutputBuffer (#58757)
Removed concurrency lock: Remove concurrency lock for better performance (#56798)

🔨 Fixes

Fixes to Unique: Fix support of list types for Unique aggregator (#58916)
Parquet NaN fix: Fix reading from written parquet for numpy with NaNs (#59172)
Hash Shuffle empty block: Fix empty block sort in hash shuffle operator (#58836)
Hive partitioning pushdown: Fix pushdown optimizations with Hive partitioning (#58723)
Object Store usage reporting: Fix obj_store_mem_max_pending_output_per_task reporting (#58864)
Pyarrow FileSystem serialization fix: Handle filesystem serialization issue in get_parquet_dataset (#57047)
Azure UC SAS: Handle Azure UC user delegation SAS (#59393)
Async UDF Thread Cleanup: Close threads from async UDF after actor died (#59261)
Object Locality Default: Default return 0s for object locality instead of -1s (#58754)

📖 Documentation

Added contributing guide to Ray Data documentation (#58589)
Added download expression to key user journeys in documentation (#59417)
Added Kafka user guide (#58881)
Added unstructured data templates from Ray Summit 2025 (#57063)
Improved instructions for reading Hugging Face datasets (#58492, #58832)
Refined batch-format guidance in docs (#58971)
Exposed vision_preprocess and vision_postprocess in VLM docs (#59012)
Added upgrading huggingface_hub instruction (#59109)
Added scaling out expensive collation functions doc (#58993)

Ray Serve

🎉 New Features

Deployment topology visibility. Exposes deployment dependency graphs in Serve REST API, allowing users to visualize and understand the DAG structure of their applications. (#58355)
External autoscaler integration. Adds external_scaler_enabled flag to application config, enabling third-party autoscalers to control replica counts. (#57727, #57698)
Node rank and local rank support. Extends replica rank system to track node-level and per-node local ranks, enabling better distributed serving coordination for multi-node deployments. (#58477, #58479)
Custom batch size function. Allows users to define custom functions for computing logical batch sizes in @serve.batch, useful when batch items have varying weights (e.g., token counts in LLM inference). (#59059)
Stateful application-level autoscaling. Adds policy state persistence for custom autoscaling policies, allowing policies to maintain state across control-loop iterations. (#59118)
New autoscaling, batching, and routing metrics. Adds Prometheus metrics for autoscaling decisions (ray_serve_deployment_target_replicas, ray_serve_autoscaling_decision_replicas), batching statistics, and router queue latency for improved observability. (#59220, #59232, #59233)

💫 Enhancements

Smarter downscaling behavior. Prioritizes stopping most recently scaled-up replicas during downscale, preserving long-lived replicas that are optimally placed and fully warmed up. (#52929)
Autoscaling performance optimizations. Short-circuits metric aggregation for single time series cases (O(n log n) → O(1)) and lazily evaluates expensive autoscaling context fields to reduce controller CPU usage. (#58962, #58963)
Route matching cleanup. Removes redundant route matching logic from replicas since correct route values are now included in RequestMetadata. Also allows multiple methods (GET, PUT) corresponding to a route. (#58927)
Deployment wrapper metadata preservation. Wrapper classes from decorators like @ingress now preserve original class metadata (__qualname__, __module__, __doc__, __annotations__). (#58478)
Improved type annotations. Enhances generic type annotations on DeploymentHandle, DeploymentResponse, and DeploymentResponseGenerator for better IDE support and type inference. Adds .result() stub to DeploymentResponseGenerator to fix static typing errors. (#59363, #58522)

🔨 Fixes

YAML serialization for autoscaling enums. Fixes RepresenterError when using serve build with AggregationFunction enum values in autoscaling config. (#58509)
Autoscaling context timestamp fix. Correctly sets last_scale_up_time and last_scale_down_time on autoscaling context. (#59057)
Deadlock in chained deployment responses. Fixes hang when awaiting intermediate DeploymentResponse objects in a chain of deployment calls from different event loops. (#59385)
FastAPI class-based view inheritance. Fixes make_fastapi_class_based_view to properly handle inherited methods. (#59410)

📖 Documentation

Async I/O best practices guide. New documentation covering async programming patterns and best practices for Ray Serve deployments. (#58909)
Replica scheduling guide. New documentation covering compact scheduling, placement groups, custom resources, and guidance on when to use each feature. (#59114)

Ray Train

🎉 New Features

Worker Placement with Label Selectors: Added label_selector to ScalingConfig. This allows users to control worker placement by targeting specific labeled nodes in the cluster. (#58845, #59414)
Multihost JaxTrainer on GPU: Introduced support for JaxTrainer running on GPU machines. (#58322)
Checkpoint Consistency Modes: Added CheckpointConsistencyMode to get_all_reported_checkpoints, providing options for handling checkpoint retrieval consistency. (#58271)
Per-Dataset Execution Options: DataConfig now supports setting execution_options on a per-dataset basis for finer-grained control over data loading. (#58717)

💫 Enhancements

Nested Metrics Support: Result.get_best_checkpoint now supports nested metrics, allowing for more flexible metric tracking and checkpoint selection. (#58537)
Non-Blocking Checkpoint Retrieval: get_all_reported_checkpoints no longer blocks when only metrics are reported. (#58870)
Improved Resource Cleanup: Implemented eager cleanup of data resources and placement groups upon training run failures or aborts, preventing resource leaks. (#58325, #58515)

🔨 Fixes

MLflow Compatibility: Updated setup_mlflow API to ensure full compatibility with Ray Train V2. (#58705)
Validation for Checkpoint Uploads: A ValueError is now raised if checkpoint_upload_fn fails to return a valid checkpoint. (#58863)

📖 Documentation

New API Documentation: Added comprehensive documentation for the ray.train.get_all_reported_checkpoints method. (#58946)

Ray Tune

💫 Enhancements:

Nested Metrics Support: Result.get_best_checkpoint now supports nested metrics, allowing for more flexible metric tracking and checkpoint selection. (#58537)

Ray LLM

💫 Enhancements

Cloud filesystem restructuring with provider-specific implementations (#58469)
Bump transformers to 4.57.3 (#58980)
Ray Data LLM config refactor (#58298)
Update vllm_engine.py to check for VLLM_USE_V1 attribute (#58820)
Infer VLLM_RAY_PER_WORKER_GPUS from fractional placement-group bundles automatically (#58949)

🔨 Fixes

Fix LLM DP release test configuration (#59090)

Ray RLlib

🎉 New Features

DreamerV3: allow num_env_runners \> 0 (#58495)

💫 Enhancements

🔥 MetricsLogger tweaks+ Stats rewrite (#56838)
move restart message into EnvRunner (#56750)
make “Footsies” less verbose (optionally) (#58939)
update an AlgorithmConfig deprecated argument with incorrect behavior/semantics (#59138)
Examples/docs cleanup:
- merge tuned examples into examples/ (#58893)
- move old API examples (#59159)
- move example run scripts (#59160)
- remove Torch 2.x doc tied to removed benchmarks (#59173)
- remove rllib/benchmark(s) folder from RLlib directory (#59158)
Testing / CI & infra cleanup (part of a larger effort to organize + harden RLlib testing):
- clean up tests folder layout in favor of /component/tests (#58890)
- re-enable and fix nightly tests for APPO on Atari and MuJoCo (#58853)
- re-enable all RLlib doctests (#58974)
- add pytest reporting hook (pytest_runtest_makereport) across tests (#59003)
- add/enable RLlib Py3.10 CI lane (#59226)
- fix as-release-test silently failing (#59386)
- fix recursive imports in old test-utils location (#59435)
- Remove asv.conf.json (#58934)
- Update requirement for byod_rllib.sh (#59157)

🔨 Fixes

Fix custom model-config mismatch between EnvRunner and Learner (#58739)
MultiAgentEnvRunner: prevent double-calling connectors (#58931)
Error handling: log or raise when a case is not fully handled (#58889)
Error handling: error out when data cannot be loaded (#59002)
Assorted RLlib bugfixes (#59386)

📖 Documentation

Update APPO paper reference to link to IMPACT paper (#58935)

Ray Core

🎉 New Features

Support zero-copy serialization for read-only PyTorch tensors via RAY_ENABLE_ZERO_COPY_TORCH_TENSORS (#57639)
Add .rayignore file support for controlling cluster uploads (#58500)
Improve large-scale resource view synchronization through sync message batching (#57641)
Autoscaler with cloud resource availability awareness (#58623)
Token authentication UX improvements with new AuthenticationError exception (#58737)
Support X-Ray-Authorization fallback header for auth token in dashboard (#58819)

💫 Enhancements

Limit core worker gRPC reply threads to 2 by default via RAY_core_worker_num_server_call_thread (#58771)
Make accessor node address and liveliness cache thread safe (#58947)
Create OtlpGrpcMetricExporter wrapper to log export failures (#58929)
Print detailed exception information when failing to report events (#58953)
Simplify local/global GC logic (#58671)
Surface correct error message when get_if_exists=True for actor lookup (#58628)
Throw AuthenticationError from Python for token loading errors (#59031)
Use secrets.token_hex(32) to generate auth tokens (#58818)
Remove AUTH_MODE=token check in get-auth-token CLI (#58848)
Introduce core chaos network release tests (#58868)

🔨 Fixes

Fix grpc_authentication_server_interceptors streaming response handling (#59104)
Fix handle leak in IsProcessAlive on Windows (#59106)
Fix counter metric default branch for RAY_enable_open_telemetry (#59095)
Fix leaking metric recorder in tests (#58952)
Fix crash when using JVM HDFS by adding RAY_DISABLE_FAILURE_SIGNAL_HANDLER option (#58984)
Fix heap corruption in RayletClient causing driver crash (use-after-free) (#58660)
Use shared_ptr for pins_in_flight_ to prevent use-after-free (#58744)
Remove deprecated add_command_alias (#58719)
Remove cluster_full_of_actors_detected_* fields (unused in autoscaler v2) (#59052)

📖 Documentation

Add token-auth.md documentation page (#58829)
Update KubeRay authentication guide to use native Ray token authentication (#58729)

Dashboard

💫 Enhancements

Add time_to_first_batch and get_ref_bundles metrics to data dashboard (#58912)
Update Ray Data histograms to show percentiles grouped by operator (#58650)

Ray Wheels and Images

Upgraded rich, cupy-cuda12x, and memray (#58983)
Upgraded lxml to 6.0.2 (#58808)
Upgraded requests from 2.32.3 to 2.32.5 (#58724)
Added openlineage-python in the dependency set (#58724)

Thanks

Thank you to everyone who contributed to this release!
@xinyuangui2, @harshit-anyscale, @Sparks0219, @israbbani, @siyuanfoundation, @robertnishihara, @thomasdesr, @spencer-p, @aslonnie, @ZacAttack, @soodoshll, @marosset, @simeetnayan81, @soffer-anyscale, @abrarsheikh, @400Ping, @richo-anyscale, @as-jding, @rueian, @kshanmol, @yancanmao, @zzchun, @coqian, @matthewdeng, @Future-Outlier, @YoussefEssDS, @ykdojo, @pseudo-rnd-thoughts, @lowdy1, @ArturNiederfahrenhorst, @myandpr, @komikndr, @machichima, @RisinT96, @curiosity-hyf, @alanwguo, @CaiZhanqi, @Aydin-ab, @MengjinYan, @suzuri-lollipop, @jeffreyjeffreywang, @rushikeshadhav, @alexeykudinkin, @meAmitPatil, @zcin, @teddygood, @elliot-barn, @dayshah, @srinathk10, @XLC127, @simonsays1980, @kevin85421, @bveeramani, @kunling-anyscale, @khluu, @andrew-anyscale, @KaisennHu, @kouroshHakha, @ryankert01, @pavitrabhalla, @jjyao, @dragongu, @SolitaryThinker, @justinrmiller, @wxwmd, @Haustle-v, @TimothySeah, @goutamvenkat-anyscale, @liulehui, @raulchen, @HassamSheikh, @Priya-753, @vaishdho1, @dancingactor, @daiping8, @eloaf, @JasonLi1909, @rayci-bot, @richardliaw, @SheldonTsen, @Yicheng-Lu-llll, @ktyxx, @pschmutz, @iamjustinhsu, @ahao-anyscale, @cem-anyscale, @eicherseiji, @edoakes, @rajeshg007, @arki05, @andrewsykim, @nrghosh, @ryanaoleary, @kyuds, @Daraan, @can-anyscale, @sampan-s-nayak, @xyuzh, @owenowenisme

Ray-2.51.2

2025-11-29T00:40:40Z

Fix for CVE-2025-62593: reject Sec-Fetch-* other browser-specific headers in dashboard browser rejection logic

Ray-2.52.1

2025-11-29T00:42:28Z

More robust handling for CVE-2025-62593: test for more browser-specific headers in dashboard browser rejection logic

Ray-2.52.0

2025-12-03T21:24:10Z

Release Highlights

Ray Core:

End of Life for Python 3.9 Support: Ray will no longer be releasing Python 3.9 wheels from now on.
Token authentication: Ray now supports built-in token authentication across all components including the dashboard, CLI, API clients, and internal services. This provides an additional layer of security for production deployments to reduce the risk of unauthorized code execution. Token authentication is initially off by default. For more information, see: https://docs.ray.io/en/latest/ray-security/token-auth.html

Ray Data:

We’ve added a number of improvements for Iceberg, including upserts, predicate and projection pushdown, and overwrite.
We’ve added significant improvements to our expressions framework, including temporal, list, tensor, and struct datatype expressions.

Ray Libraries

Ray Data

🎉 New Features:

Added predicate pushdown rule that pushes filter predicates past eligible operators (#58150, #58555)
Iceberg support for upsert tables, schema updates, and overwrite operations (#58270)
Iceberg support for predicate and projection pushdown (#58286)
Iceberg write datafiles in write() then commit (#58601)
Enhanced Unity Catalog integration (#57954)
Namespaced expressions that expose PyArrow functions (#58465)
Added version argument to read_delta_lake (#54976)
Generator UDF support for map_groups (#58039)
ApproximateTopK aggregator (#57950)
Serialization framework for preprocessors (#58321)
Support for temporal, list, tensor, and struct datatypes (#58225)

💫 Enhancements:

Use approximate quantile for RobustScaler preprocessor (#58371)
Map batches support for limit pushdown (#57880)
Make all map operations zero-copy by default (#58285)
Use tqdm_ray for progress reporting from workers (#58277)
Improved concurrency cap backpressure tuning (#58163, #58023, #57996)
Sample finalized partitions randomly to avoid lens effect (#58456)
Allow file extensions starting with '.' (#58339)
Set default file_extensions for read_parquet (#56481)
URL decode values in parse_hive_path (#57625)
Streaming partition enforces row_num per block (#57984)
Streaming repartition combines small blocks (#58020)
Lower DEFAULT_ACTOR_MAX_TASKS_IN_FLIGHT_TO_MAX_CONCURRENCY_FACTOR to 2 (#58262)
Set udf-modifying-row-count default to false (#58264)
Cache PyArrow schema operations (#58583)
Explain optimized plans (#58074)
Ranker interface (#58513)

🔨 Fixes:

Fixed renamed columns to be appropriately dropped from output (#58040, #58071)
Fixed handling of renames in projection pushdown (#58033, #58037)
Fixed broken LogicalOperator abstraction barrier in predicate pushdown rule (#58683)
Fixed file size ordering in download partitioning with multiple URI columns (#58517)
Fixed HTTP streaming file download by using open_input_stream (#58542)
Fixed expression mapping for Pandas (#57868)
Fixed reading from zipped JSON (#58214)
Fixed MCAP datasource import for better compatibility (#57964)
Avoid slicing block when total_pending_rows < target (#58699)
Clear queue for manually marked execution_finished operators (#58441)
Add exception handling for invalid URIs in download operation (#58464)
Fixed progress bar name display (#58451)

📖 Documentation:

Documentation for Ray Data metrics (#58610)
Simplify and add Ray Data LLM quickstart example (#58330)
Convert rST-style to Google-style docstrings (#58523)

🏗 Architecture:

Removed stats update thread (#57971)
Refactor histogram metrics (#57851)
Revisit OpResourceAllocator to make data flow explicit (#57788)
Create unit test directory for fast, isolated tests (#58445)
Dump verbose ResourceManager telemetry into ray-data.log (#58261)

Ray Train

🎉 New Features:

Result::from_path implementation in v2 (#58216)

💫 Enhancements:

Exit actor and log appropriately when poll_workers is in terminal state (#58287)
Set JAX_PLATFORMS environment variable based on ScalingConfig (#57783)
Default to disabling Ray Train collective util timeouts (#58229)
Add SHUTTING_DOWN TrainControllerState and improve logging (#57882)
Improved error message when calling training function utils outside Ray Train worker (#57863)
FSDP2 template: Resume from previous epoch when checkpointing (#57938)
Clean up checkpoint config and trainer param deprecations (#58022)
Update failure policy log message (#58274)

📖 Documentation:

Ray Train Metrics documentation page (#58235)
Local mode user guide (#57751)
Recommend tree_learner="data_parallel" in examples for distributed LightGBM training (#58709)

Ray Serve

🎉 New Features:

Custom request routing with runtime environment support. Users can now define custom request router classes that are safely imported and serialized using the application's runtime environment, enabling advanced routing logic with custom dependencies. (#56855)
Custom autoscaling policies with enhanced logging. Deployment-level and application-level autoscaling policies now display their custom policy names in logs, making it easier to debug and monitor autoscaling behavior. (#57878)
Audio transcription support in vLLM backend. Ray Serve now supports transcription tasks through the vLLM engine, expanding multimodal capabilities. (#57194)
Data parallel attention public API. Introduced a public API for data parallel attention, enabling efficient distributed attention mechanisms for large-scale inference workloads. (#58301)
Route pattern tracking in proxy metrics. Proxy metrics now expose actual route patterns (e.g., /api/users/{user_id}) instead of just route prefixes, enabling granular endpoint monitoring without high cardinality issues. Performance impact is minimal (~1% RPS decrease). (#58180)
Replica dependency graph construction. Added list_outbound_deployments() method to discover downstream deployment dependencies, enabling programmatic analysis of service topology for both stored and dynamically-obtained handles. (#58345, #58350)
Multi-dimensional replica ranking. Introduced ReplicaRank schema with global, node-level, and local ranks to support advanced coordination scenarios like tensor parallelism and model sharding across nodes. (#58471, #58473)
Proxy readiness verification. Added a check to ensure proxies are ready to serve traffic before serve.run() completes, improving deployment reliability. (#57723)
IPv6 socket support. Ray Serve now supports IPv6 networking for socket communication. (#56147)

💫 Enhancements:

Selective throughput optimization flag overrides. Users can now override individual flags set by RAY_SERVE_THROUGHPUT_OPTIMIZED without manually configuring all flags, improving flexibility for performance tuning. (#58057)
OpenTelemetry metrics enabled by default. Ray now uses OpenTelemetry as the default metrics backend, with updated metric names (ray_serve_*) and improved observability infrastructure. (#56432)
Cleaner long-poll communication. Removed actor handles from RunningReplicaInfo objects passed in long-poll updates, avoiding complex reference counting patterns. (#58174)
Improved replica config handling. Excluded IMPLICIT_RESOURCE_PREFIX from ReplicaConfig.ray_actor_options to prevent internal resource annotations from leaking into user-visible configurations. (#58275)
Custom autoscaling telemetry. Added telemetry tracking for custom autoscaling policy usage. (#58336)
Proxy target group control. Added from_proxy_manager argument to get_target_groups() for finer control over returned routing targets. (#57620)

🔨 Fixes:

Fixed default deployment name in async inference. Corrected the default deployment name which was changed to _TaskConsumerWrapper during async inference implementation. (#57664)
Fixed proxy location handling in CLI and Python API. serve run now respects proxy_location from config files instead of hardcoding EveryNode, and serve.start() no longer defaults to HeadOnly when http_options are provided without an explicit location. (#57622)
Fixed deprecated Stable Diffusion model in example. Updated documentation example to use a current model after stabilityai/stable-diffusion-2 was deprecated on Hugging Face. (#58609)

📖 Documentation:

KV-cache offloading user guide. Added comprehensive documentation for KV-cache offloading in LLM deployments. (#58025)
Model loading documentation. Documented best practices and options for loading models in Ray Serve. (#57922)
Cross-node tensor/pipeline parallelism examples. Added examples and documentation for running TP/PP across multiple nodes. (#57715)
Data parallel attention documentation. Created user guide for data parallel attention with architecture diagrams. (#58301, #58543)
Custom autoscaling policy examples. Added missing imports and improved clarity in autoscaling policy examples. (#57896, #58170)
Async inference documentation improvements. Added notes about task consumer replica configurations and fixed the end-to-end example. (#58493)
Callback documentation. Added documentation for using callbacks in Ray Serve. (#58713)
Monitoring and troubleshooting improvements. Enhanced monitoring section with links to Anyscale troubleshooting resources. (#58472)
Minor documentation fixes. Fixed spelling errors and improved docstring alignment. (#58172, #58233)

🏗 Architecture refactoring:

Replica rank management refactoring. Extracted generic RankManager class with type-safe ReplicaRank representation, creating a cleaner foundation for future multi-level rank support. (#58471, #58473)

Ray Tune

💫 Enhancements:

Updated jobs test to use tune module (#57995)
Add pydantic to Ray Tune requirements (#58354)

RLlib

🎉 New Features:

Support for vectorize modes in SingleAgentEnvRunner.make_env (#58410)
Support for composed spaces in Offline RL (#58594)
Enhanced support for complex observations in SingleAgentEpisode (#57017)
Prometheus metrics support for selected components (#57932)

💫 Enhancements:

Improve test_single_agent_env_runner to prevent flaky tests (#58397)
LINT improvements with enabled ruff imports across multiple modules (#56737, #56734, #56741, #56742, #56744, #56746)

🔨 Fixes:

Resolve bug that fails to propagate model_config to MultiAgentRLModule instances (#58243)
Fixed access to self._minibatch_size (#58595)
Broken restore from remote - Add missing FileSystem argument (#58324)
Fixed deterministic sampling and training documentation link (#58494)
Corrected typo in pyspiel import error message (#54618)

📖 Documentation:

Add reinforcement learning example illustrating GPU-to-GPU RDT and GRPO (#57961)

Ray Core

🎉 New Features:

Token-based authentication across all Ray components (#58046, #58047, #58176, #58209, #58276, #58281, #58308, #58333, #58368, #58395, #58405, #58408, #58424, #58557, #57835, #58566, #58591)
OpenTelemetry enabled by default for improved observability (#56432)
Fallback strategy scheduling logic (#56369)
TPU utility functions to support slice placement groups (#56723)
Exponential backoff for retryable gRPCs (#56568)
Option for in-flight RPC failure injection (#58512)
Release test to simulate network transient errors via iptables (#58241)
Nightly release test with cross-AZ fault injection (#57579)
Owned object spill metrics (#57870)
Monitoring in raylet for resource view (#58382)
IPv6 support for sockets (#56147)

💫 Enhancements:

Fault-tolerant RPCs: KillActor, CancelRemoteTask, NotifyGCSRestart, and ReleaseUnusedBundles (#57648, #57945, #57965)
Use graceful actor shutdown when GCS polling detects actor ref deleted (#58605)
Use graceful shutdown path when actor OUT_OF_SCOPE (del actor) (#57090)
Improved actor kill logs (#58544)
Scheduling detached actor with placement group not recommended (#57726)
Better handling of detached actor restarts (#57931)
Enhanced ray.get thread safety (#57911)
Making concurrent ray.get requests for the same object thread-safe (#58606)
Move request ID creation to worker to address plasma get perf regression (#58390)
Make GlobalState lazy initialization thread-safe (#58182)
Reporter agent can get PID via RPC to raylet (#57004)
Add tee logging for subprocess exit codes in ray start --block (#57982)
Add entrypoint log for jobs (#58300)
Cleaner error message for exceeding list actors limit (#58255)
Clean up NODE_DIED task error message (#58638)
Improved histogram metrics midpoint calculation (#57948)
Migrated from STATS to metric interface in RPC components (#57926)
Kill STATS in core worker component (#58060)
Kill STATS in object manager component (#57974)
Improve scheduler_placement_time_s metric (#58217)
Refactor OpenTelemetry environment variable handling (#57910)
Add option to disable OpenTelemetry SDK error logs (#58257)
Improved cgroups support (#57776, #57864, #57731, #58017, #58028, #58059, #58064, #58577)
Use GetNodeAddressAndLiveness in raylet client pool (#58576)
Ray Direct Transport improvements with NIXL integration (#57671, #58550, #58548, #56783, #58263)
Fix symmetric-run (#58337)
Make worker connection timeout parameters configurable (#58372)
Define env for controlling UVloop (#58442)
Allow 60 seconds for dashboard to start (#58341)
Report driver stats (#58045)
Fix idle node termination on object pulling (#57928)
Check if temp_dir is subdir of virtualenv to prevent runtime virtualenv problems (#58084)

🔨 Fixes:

Fixed use-after-free in RayletClient (#58747)
Fixed deadlock when cancelling stale requests on in-order actors (#57746)
Fixed "RayEventRecorder::StartExportingEvents() should be called only once" error (#57917)
Fixed raylet shutdown races (#57198)
Fixed incorrect usage of gRPC streaming API in ray syncer (#58307)
Fixed log monitor seeking bug after log rotation (#56902)
Fixed idempotency issues in RequestWorkerLease for scheduled leases (#58265)
Fixed RAY_CHECK(inserted) inside reference counter (#58092)
Fixed static type hints for ActorClass when setting options (#58439)
Fixed exception type for accelerator ID visibility check (#58269)
Fixed transport type handling in DAG node initialization (#57987)
Fixed RAY_NODE_TYPE_NAME handling when autoscaler is in read-only mode (#58460)
Ensure client_call_manager_ outlives metrics_agent_client_ in core worker (#58315)
Fixed header validation in dashboard tests (#58648)
Validation of Ray-on-Spark-on-YARN mode to enable it to run (#58335)

📖 Documentation:

Fix pattern_async_actor demo typo (#58486)
Add limitations of RDT documentation (#58063)
Add actor+job+node event to ray event export documentation (#57930)
Remove implementation details from get_runtime_context docstring (#58212)
Improved monitoring section with links (#58472)

🏗 Architecture:

Refactor ActorInfoAccessor in gcs_client to be mockable (#57241)
Refactor reference_counter out of memory store and plasma store (#57590)
Remove reference counter mock for real reference counter in testing (#57178)
Split raylet cython file into multiple files (#56575)
Move ray_syncer to top level directory (#58316)
Move python_callbacks to common (#57909)
Consolidate find_free_port to network_utils (#58304)
Implement event merge logic at export time (#58070)
Feature flag for enabling ray export event (#57999)
Add comments explaining ray_syncer_ channels in Raylet (#58342)
Integration tests for task event generation (#57636)

Dashboard

💫 Enhancements:

Added percentage usage graphs for resources (#57549)
Sub-tabs with full Grafana dashboard embeds on Metrics tab (#57561)
Added queued blocks to operator panels (#57739)
Improved operator metrics logging (#57702)
Make do_reply accept status_code instead of success bool (#58384)
Add denial of fetch headers (#58553)

🔨 Fixes:

Fixed broken Ray Data per node metrics due to unsupported operator filter (#57970)
Filtered out ANSI escape codes from logs (#53370)

📖 Documentation:

Expose dashboard URL when deploying on Yarn using Skein (#57793)

Autoscaler + KubeRay

🎉 New Features:

KubeRay autoscaling support with top-level Resources and Labels fields (#57260)
Bundle label selector support in request_resources SDK (#54843)

💫 Enhancements:

Azure VM launcher release test (#57921)
Azure CLI added to base-extra image (#58012)

📖 Documentation:

Label selector guide (#58157)
Add minimum version requirement on kai-scheduler (#58161)
Mention RayJob gang scheduling for Yunikorn (#58375)
Add Volcano RayJob gang scheduling example (#58320)
Add KAI scheduler integration documentation (#54857)
Kuberay sidecar mode (#58273)
Update RayJob documentation with new DeletionStrategy (#58306)
Add guidance for RayService initialization timeout (#58238)
Update version to 1.5.0 (#58452)
Add output example of CLI commands (#58078)
Fix invalid syntax in label_selector (#58352)

Thank You to all the Contributors!
@marosset, @curiosity-hyf, @bveeramani, @Future-Outlier, @saihaj, @ZacAttack, @ArthurBook, @crypdick, @Aydin-ab, @elliot-barn, @Kunchd, @justinvyu, @jjyao, @gangsf, @sunsetxh, @Daraan, @justinyeh1995, @MatthewCWeston, @kyuds, @daiping8, @sauravvenkat, @omatthew98, @CowKeyMan, @morotti, @israbbani, @goutamvenkat-anyscale, @fscnick, @Zakelly, @xyuzh, @kouroshHakha, @owenowenisme, @Qiaolin-Yu, @czgdp1807, @shen-shanshan, @wph95, @iamjustinhsu, @MengjinYan, @jugalshah291, @Yicheng-Lu-llll, @ryanaoleary, @nadongjun, @xinyuangui2, @ideal, @my-vegetable-has-exploded, @lucaschadwicklam97, @tianyi-ge, @ahao-anyscale, @abrarsheikh, @Blaze-DSP, @rueian, @thomasdesr, @CaiZhanqi, @harshit-anyscale, @jeffreyjeffreywang, @TimothySeah, @codope, @sampan-s-nayak, @andrewsykim, @xingsuo-zbz, @aslonnie, @OneSizeFitsQuorum, @ryankert01, @Sparks0219, @soffer-anyscale, @akyang-anyscale, @alanwguo, @chrisfellowes-anyscale, @richo-anyscale, @alexeykudinkin, @JasonLi1909, @ruisearch42, @EkinKarabulut, @MarcoGorelli, @SolitaryThinker, @srinathk10, @dayshah, @richardliaw, @pseudo-rnd-thoughts, @win5923, @axreldable, @matthewdeng, @ArturNiederfahrenhorst, @can-anyscale, @khluu, @landscapepainter, @kevin85421, @seanlaii, @edoakes, @nrghosh, @eicherseiji, @Artimislyy, @cem-anyscale, @coqian, @chiayi, @liulehui

Ray-2.51.1

2025-11-21T18:17:24Z

Reuse previous metadata if transferring the same tensor list with nixl (#58309)

Ray-2.51.0

2025-10-29T22:07:22Z

Release Highlights

Ray Train:

Ray Train v2 is now enabled by default! Ray Train v2 provides usability and stability improvements, as well as new features. For more details, see the REP and Migration Guide. To disable Ray Train v2, set the environment variable RAY_TRAIN_V2_ENABLED=0.

Ray Serve:

Application-level autoscaling: Introduces custom autoscaling policies that operate across all deployments in an application, enabling coordinated scaling decisions based on aggregate metrics. This is a significant advancement over per-deployment autoscaling, allowing for more intelligent resource management at the application level.
Enhanced autoscaling capabilities with replica-level metrics: Wires up AutoscalingContext with total_running_requests, total_queued_requests, and total_num_requests, plus adds support for min, max, and time-weighted average aggregation functions. These improvements give users fine-grained control to implement sophisticated custom autoscaling policies based on real-time workload metrics.

Ray Libraries

Ray Data

🎉 New Features:

Added enhanced support for Unity Catalog integration (#57954, #58049)
New expression evaluator infrastructure for improved query optimization (#57778, #57855)
Support for SaveMode in write operations (#57946)
Added approximate quantile aggregator (#57598)
MCAP datasource support for robotics data (#55716)
Callback-based stat computation for preprocessors and ValueCounter (#56848)
Support for multiple download URIs with improved error handling (#57775)

💫 Enhancements:

Improved projection pushdown handling with renamed columns (#58033, #58037, #58040, #58071)
Enhanced hash-shuffle performance with better retry policies (#57572)
Streamlined concurrency parameter semantics (#57035)
Improved execution progress rendering (#56992)
Better handling of empty columns in pandas blocks (#57740)
Enhanced support for complex data types and column operations (#57271)
Reduced memory usage with improved streaming generator backpressure (#57688)
Enhanced preemption testing and utilities (#57883)
Improved Download operator display names (#57773)
Better handling of variable-shaped tensors and tensor columns (#57240)
Optimized aggregator execution with out-of-order processing by default (#57753)

🔨 Fixes:

Fixed renamed columns to be appropriately dropped from output (#58040, #58071)
Fixed handling of renames in projection pushdown (#58033, #58037)
Fixed vLLMEngineStage field name inconsistency for images (#57980)
Fixed driver hang during streaming generator block metadata retrieval (#56451)
Fixed retry policy for hash-shuffle tasks (#57572)
Fixed prefetch loop to avoid blocking on fetches (#57613)
Fixed empty projection handling (#57740)
Fixed errors with concatenation of mixed pyarrow native and extension types (#56811)

📖 Documentation:

Updated document embedding benchmark to use canonical Ray Data API (#57977)
Improved concurrency-related documentation (#57658)
Updated preprocessing and data handling examples

Ray Train

🎉 New features

Turn on Train v2 by default (#57857)
Top-level ray.train aliases for public APIs (#57758)

💫 Enhancements

Raise clear errors when mixing v1/v2 APIs (#57570)
JAX backend: add jax.distributed.shutdown() for JaxBackend (#57802)
Update TrainingFailedError module (#57865)
Improve deprecation handling when ray.train methods are called from ray.tune (#57810)
Enable deprecation warnings for legacy XGBoost/LightGBM trainers (#57280)

🔨 Fixes

Fix ControllerError triggered by after_worker_group_poll_status errors (#57869)
Fix iter_torch_batches use of ray.train.torch.get_device outside Train (#57816)
Fix exception-queue race condition in ThreadRunner (#57249)

📖 Documentation

Add validation and details to checkpoint docs (#57065)

🏗 Architecture / tests

Enable Train v2 across test suites; migrate remaining tests and isolate/disable stragglers (#56868, #57256, #57534, #57722, #57764)
Isolate circular-dependency tests and resolve circular imports (#57710, #56921)
Replace Checkpoint Manager Pydantic v2 APIs with v1 (#57147)
Bump test timeouts (test_util, torch_trainer) (#57939, #57873)

Ray Tune

💫 Enhancements:

Updated release tests to import from tune (#57956)
Better integration with Train V2 backend

Ray Serve

🎉 New Features:

Application-level autoscaling. Introduces support for custom autoscaling policies that operate across all deployments in an application, enabling coordinated scaling decisions based on aggregate metrics. (#57535, #57548, #57637, #57756)
Autoscaling metrics aggregation functions. Adds support for min, max, and time-weighted average aggregation over timeseries data, providing more flexible autoscaling control. (#56871)
Enhanced autoscaling context with replica-level metrics. Wires up AutoscalingContext constructor arguments to expose total_running_requests, total_queued_requests, and total_num_requests for use in custom autoscaling policies. (#57202)
Multiple task consumers in a single application. Ray Serve applications can now run multiple task consumer deployments concurrently. (#56618)

💫 Enhancements:

Reconfigure invoked on replica rank changes. The reconfigure method now receives both user_config and rank parameters when ranks change, enabling replicas to adapt their configuration dynamically. (#57091)
Celery adapter configuration improvements. Added default serializer and new configuration fields to enhance Celery integration flexibility. (#56707)
AutoscalingContext promoted to public API. The autoscaling context is now officially part of the public API with comprehensive documentation. (#57600)
Async inference telemetry. Added telemetry tracking to monitor the number of replicas using asynchronous inference. (#57665)
Rank logging verbosity reduced. Changed seven rank-related INFO logs to DEBUG level, reducing log noise during normal operations. (#57831)
Controller logging optimized. Removed expensive debug logs from the controller that were costly in large clusters. (#57813)

🔨 Fixes:

Max constructor retry count test fixed for Windows. Adjusted test resource requirements to account for Windows process creation overhead compared to Linux forking. (#57541)
Streaming test stability improvements. Added synchronization mechanisms to prevent chunk coalescing and rechunking, eliminating test flakiness. (#57592, #57728)
Autoscaling test deflaking. Fixed race conditions in application-level autoscaling tests and removed flaky min aggregation test scenario. (#57784, #57967)
State API usage test corrected. Fixed a unit test that was broken but not running in CI. (#56948)
Controller recovery logging condition fixed. Updated test condition to properly verify debug and JSON logs after controller recovery. (#57568)

📖 Documentation:

Custom autoscaling documentation. Added comprehensive guide for implementing custom autoscaling policies with examples and best practices. (#57600)
Replica ranks documentation. Documented the replica rank feature, including how ranks are assigned and how to use them in reconfigure methods. (#57649)
Application-level autoscaling guide. Added documentation explaining how to configure and use application-level autoscaling policies. (#57756)
Autoscaling documentation improvements. Updated serve autoscaling docs with clearer explanations and examples. (#57652)
Performance flags documentation. Documented performance-related configuration flags for Ray Serve. (#57845)
Metrics documentation fix. Corrected ray_serve_deployment_queued_queries metric name discrepancy in documentation. (#57629)
AutoscalingContext import added to examples. Fixed missing import statement in custom autoscaling policy example. (#57876)
App builder guide typo corrected. Fixed command syntax error in typed application builder example. (#57634)
Celery filesystem broker note. Added warning about using filesystem as a broker in Celery workers. (#57686)
Async inference alpha stage warning. Added notice that async inference is in alpha stage. (#57268)

🏗 Architecture refactoring:

Autoscaling control moved to application state. Migrated autoscaling control loop from deployment state to application state, preparing for application-level autoscaling. (#57548)
Async capability enum removed. Cleaned up unused async capability enum from codebase. (#57666)

Ray Serve/Data LLM

🎉 New Features:

Updated vLLM to 0.11.0 and Nixl to 0.6.0 (#57201)
Video processor support for multimodal pipelines (#56785)
Enhanced callback API for engine customization (#57257)
Unified and extended builder configuration for LLM deployments (#57724)

💫 Enhancements:

Protocol-based typing improvements and cleaner inheritance structure (#57743)
Better engine metrics enabled by default (#57615)
Simplified NIXL dependency management in ray-llm images (#57706)
Per-stage map kwargs for LLM processor preprocessing/postprocessing (#57826)
Improved architecture documentation (#57830)
Better code structure alignment with architectural design (#57889)
Enhanced multimodal support with Deepseek compatibility (#56906)

🔨 Fixes:

Fixed NIXL limitations with proper exception handling (#58159)
Improved runai_streamer for vLLM 0.10.2+ integration (#56906)

📖 Documentation:

Added comprehensive architecture documentation for Ray Serve LLM (#57830)
Reorganized LLM documentation with improved navigation (#57787)
Added benchmark page for performance reference (#57960)
Converted quick-start guide to MyST Markdown (#57782)
Better organization of Ray Serve LLM documentation (#57181?)

RLlib

🎉 New Features:

Prometheus metrics support for selected RLlib components (#57932)
Enhanced support for complex observations in SingleAgentEpisode (#57017)

💫 Enhancements:

LINT improvements with enabled ruff imports for rllib/utils (#56737)
Better type hints for learner_connector (#57673)
Improved throughput metrics to avoid biasing (#57215)

🔨 Fixes:

Fixed segment_tree.py edge case (#57599)
Fixed small bug in type hints (#57673)

Ray Core

🎉 New Features:

Enhanced Ray Direct Transport (RDT) with improved NIXL integration and garbage collection (#57671, #57603, #58159)
Cgroups support improvements with better system resource management (#57776, #57864, #57731, #58017, #58028, #58064)
Fault-tolerant RPC improvements for better distributed reliability (#57786, #57861)
Exponential backoff for retryable gRPCs (#56568)

💫 Enhancements:

Migrated from STATS to metric interface in RPC components (#57926)
Improved histogram metrics midpoint calculation (#57948)
Made FreeObjects non-fatal for better error handling (#57550)
Enhanced ReleaseUnusedBundles fault tolerance (#57786)
Made DrainRaylet and ShutdownRaylet fault tolerant (#57861)
Better error handling for metric and event exporter agent (#57925)
Improved raylet shutdown process and file organization (#57817)
Reporter agent can now get PID via RPC to raylet (#57004)
Enhanced ray.get thread safety (#57911)
Configurable proto naming during event JSON conversion (#57705)
Better handling of detached actor restarts (#57931)
Improved lease rescheduling in local lease manager during node draining (#57834)

🔨 Fixes:

Fixed "RayEventRecorder::StartExportingEvents() should be called only once" error (#57917)
Fixed deadlock when cancelling stale requests on in-order actors (#57746)
Fixed raylet shutdown races (#57198)
Fixed log monitor seeking bug after log rotation (#56902)
Deflaked multiple test suites for better CI reliability
Fixed various memory and resource management issues
Better handling of actor and task failures

📖 Documentation:

Added JaxTrainer API overview to Ray docs (#57182)
Fixed various typos and documentation issues
Updated autoscaling and system configuration guides
Enhanced SLURM documentation with symmetric-run support (#56775)

🏗 Architecture refactoring:

Dashboard API server subprocesses moved into system cgroup (#57864)
Driver moved into workers cgroup for better isolation (#57776)
Improved worker-raylet interface separation (#57804)
Better plasma store provider architecture

Dashboard

💫 Enhancements:

Added percentage usage graphs for resources (#57549)
Introduced sub-tabs with full Grafana dashboard embeds on Metrics tab (#57561)
Added queued blocks to operator panels (#57739)
Improved operator metrics logging for better clarity (#57702)
Better filtering and display in job lists

🔨 Fixes:

Fixed filtering issue in job list (#56946)
Fixed incomplete card content on overview page (#56947)
Filtered out ANSI escape codes from logs (#53370)

Autoscaler

🎉 New Features:

KubeRay autoscaling support with top-level Resources and Labels fields (#57260)
Bundle label selector support in request_resources SDK (#54843)
Application Gateway for Containers as ingress for Ray clusters on Azure

💫 Enhancements:

Azure improvements: Cleaning up extra resources (MSI, VNET, NSG) during cluster teardown (#57610)
Updated defaults for Azure cluster templates (#57716)
Better availability zone support for Azure node pools (#55532)
Hello world release tests for Azure and GCE (#57597, #57695)
Improved cluster resource state handling to fix over-provisioning (#57130)

🔨 Fixes:

Fixed autoscaler state synchronization issues (#57010)
Better handling of node state information (#57130)
Improved timeout handling for patch requests (#56605)

Thank you to everyone who contributed to this release!
Special thanks to all the contributors who helped make Ray 2.51.0 possible through bug fixes, features, documentation improvements, and testing efforts.

Ray-2.50.1

2025-10-18T19:21:43Z

Ray Core: Fix deadlock when cancelling stale requests on in-order actors (#57746)

Ray-2.50.0

2025-10-10T23:20:05Z

Release Highlights

Ray Data:
This release offers many updates to Ray Data, including:

The default shuffle strategy is now changed from sort-based to hash-based. This will result in much lower peak memory usage and improved shuffle performance for aggregations.
We’ve added a new expression API enables predicate-based filtering, UDF transformations with with_column, and column aliasing for more powerful data transformations
Ray Data LLM has a number of new enhancements for multimodal data pipelines, including multi-node tensor and pipeline parallelism support per replica and ability to share vLLM engines across processors.

Ray Core:

Alpha release of Ray Direct Transport (formerly GPU objects) - simply enable it by adding the tensor_transport parameter to the existing native Ray Core API. This keeps GPU data in GPU memory until a transfer is needed, avoiding expensive serialization and copies to and from the Ray object store. It uses efficient data transports such as collective communication libraries (GLOO or NCCL) or point-to-point RDMA (via NVIDIA’s NIXL) to transfer data directly between devices, including both CPUs and GPUs.

Ray Train:

Local mode support for multi-process training with torchrun, enhanced checkpoint management with new upload modes and validation functions

Ray Serve:

Async Inference alpha release - New Ray Serve APIs for supporting long-running asynchronous inference tasks, such as for video or large document processing. Includes capabilities for using different message brokers, adapters like celery and DLQ.
Support for replica ranks - Replica level ranks are added for supporting large-model inference use-cases such as wide Data Parallel and Expert Parallel setups.
FastAPI factory pattern support - Enables using FastAPI plugins that are not serializable via cloudpickle.
Throughput optimizations - Enable these using the RAY_SERVE_THROUGHPUT_OPTIMIZED environment variable.

RLLib:
Add StepFailedRecreateEnv exception for users with unsatisfiable environments

Ray Serve/Data LLM:

Improvements to multi node serving, loading models from remote storages, and sharing resources for efficiency (fractional gpus, sharing gpus on a data pipeline with shared stages)

Ray Libraries

Ray Data

🎉 New Features:

Expression and Filtering API: New expression API enables predicate-based filtering, UDF transformations with with_column, and column aliasing for more powerful data transformations (#56716, #56313, #56550, #55915, #55788, #56193, #56596)
Added support for projection pushdown into Parquet reads (#56500)
New download expression enables efficient loading of data from columns containing URIs with improved performance and error handling (#55824, #56462, #56294, #56852, #57146)
New explain() API provides insights into dataset execution plans (#55482)
Added streaming_train_test_split to avoid materialization for train/test splits (#56803)
Ray Data LLM:
- Enabled multi-node tensor and pipeline parallelism for LLM processing (#56779)
- Added chat_template_kwargs parameter for customizing chat templates (#56490)
- Added support for OpenAI's nested image URL format in multimodal pipelines (#56584)
- vLLM engines can now be shared across sequential processors for better resource utilization (#55179)
Enhanced Dataset.stats() output with input/output row counts per operator (#56040)
Added new metrics for task duration, inputs per task, and output blocks (#56958, #56379)
Time to first batch metric for better iteration performance monitoring (#55758)
Added type-specific aggregators for numerical, categorical, and vector columns (#56610)
Added fine-grained concurrency controls with max_task_concurrency and resource allocation options (#56370, #56381)

💫 Enhancements:

Join and shuffle improvements:
- Default shuffle strategy changed from sort-based to hash-based for better performance (#55510)
- Improved groupby performance with sort-shuffle pull-based approach (#57014)
- Improved join operations with new abstractions (#57022, #56945, #55759)
Tensor type handling improvements:
- Improved compatibility between PyArrow native types, extension types, and pandas Arrow dtypes (#57566, #57176, #57057)
- Joins now supported with list/tensor non-key columns (#5648)
- Enhanced support for variable-shaped tensor arrays with different dimensions (#57240, #56918, #56457)
- Added serialization/deserialization for PyArrow Extension Arrays (#51972)
Removing Parquet metadata fetching in ParquetDatasource (#56105)
Resource requirements (num_cpus/gpus, memory) are now top-level parameters in most APIs for easier configuration (#56419)
zip() operator now supports combining multiple datasets, not just pairs (#56524)
Concurrency parameter now accepts tuples for more flexible configuration (#55867)
Write operations now use iterators instead of accumulating blocks in memory (#57108)
Reduced memory usage for OneHotEncoder (#56565)
Reduced memory usage for schema unification (#55880)
Eliminated unnecessary block copying and double execution of arrow conversions (#56569, #56793)
Improved Parquet encoding ratio estimation (#56268)
Enabled per-block limiting for Limit operator (#55239)
Optimized schema handling with deduplication and removed unnecessary unification (#55854, #55926)
Improved issue detection with event emission instead of just logs (#55717)
Better metric organization and external queue metric handling (#55495, #56604)
New backpressure policy based on downstream processing capacity (#55463)

🔨 Fixes:

Fixed streaming executor to properly drain output queues (#56941)
Improved resource management and reservation for operators (#56319, #57123)
Fixed retry logic for hash shuffle operations (#57575)
Fix split_blocks produce empty blocks (#57085)
Initialize datacontext after setting src_fn_name in actor worker (#57117)
Fix mongo datasource collStats invocation (#57027)
Fixing empty projection handling in ParquetDataSource (#56299)
Fix UnboundLocalError when read_parquet with columns and no partitioning (#55820)
Fix high memory usage with FileBasedDatasource & ParquetDatasource when using a large number of files (#55978)
[llm] Fixed LLM processor deployment with Ray Serve (#57061)
[llm] Fixed multimodal image extraction when system prompts are absent (#56435)
Ignore metadata for pandas block (#56402)
Remove metadata for hashing + truncate warning logs (#56093)

📖 Documentation:

Error in ray.data.groupby example in docs. (#57036)
Update on ray.data.Dataset.map() type hints. (#52455)
Small typo fix. (#56560)
Fix a typo. (#56587)
Fix documentation for new execution options resource limits assignment. (#56051)
Fix broken code snippets in user guides. (#55519)
Add Autoscaling Config for Context docs. (#55712)
Make object store tuning tips consistent with other pages. (#56705)
New example of how to perform batch inference with embedding models (#56027)

Ray Train

🎉 New Features:

Local mode support for Ray Train V2
- Add local mode support to Ray Train v2 (num_workers=0). (#55487)
- Add PyTorch local mode support for multi-process training with torchrun. (#56218)
Async checkpoint and validation for Ray Train
- Add checkpoint_upload_mode to ray.train.report. (#55637)
- Add checkpoint_upload_function to ray.train.report. (#56208)
- Add validate_function and validate_config to ray.train.report. (#56360)
- Add ray.train.get_all_reported_checkpoints method. (#54555)

💫 Enhancements:

Ray Train V2 Migration
- Implement BaseWorkerGroup for V1/V2 compatibility. (#57151)
Train Controller is always actor + fix tune integration to enable this. (#55556)
Refactor AcceleratorSetupCallback to use before_init_train_context. (#56509)
Move collective implementations to train_fn_utils. (#55689)
Ray Train Framework support enhancements
- Add hf trainer support for dictionary of datasets. (#56484)
- Add usage tag key for JaxTrainer. (#55887)
Add Torch process group shutdown timeout. (#56182)
Ray Train disables blocking get inside async warning. (#56757)
ThreadRunner captures exceptions from nested threads. (#55756)
Abort reconciliation thread catches ray.util.state.get_actor exception. (#56600)
Ray Data Integration
- Minor rework of get_dataset_shard. (#55825)
- Create a deepcopy of the data context on the split coordinator process. (#56211)
- Enable debug logging; fix default actor_locality_enabled. (#56632)
Refactor call_with_retry into shared library and use it to retry checkpoint upload. (#56608)
Remove Placement Group on Train Run Abort. (#56011)

🔨 Fixes:

Fix LightGBM v2 callbacks for Tune only usage. (#57042)
Ignore tensorflow test for py312. (#56244)
Revising test_jax_trainer flaky test. (#56854)
Fix test_jax_trainer imports. (#55799)
Fix test_jax_trainer::test_minimal_multihost Flaky Test. (#56548)
Disable drop_last flag to fix division by zero in torch dataloader baselines. (#56395)
Preload a subset of modules for torch dataloader forkserver multiprocessing. (#56343)

📖 Documentation:

Add checkpoint_upload_mode to checkpoint docs. (#56860)
Add get_all_reported_checkpoints and ReportedCheckpoint to API docs. (#56174)
Fix typo for Instantiating in ray train doc. (#55826)

🏗 Architecture refactoring:

Release tests for ray train local mode. (#56862)
Migrate tune_rllib_connect_test & tune_cloud_long_running_cloud_storage to ray train v2. (#56844)
Add v2 multinode persistence release test. (#56856)
Attach a quick checkpoint when reporting metrics. (#56718)
Upgrade tune_torch_benchmark to v2. (#56804)
Move tune_with_frequent_pausing to Ray Train v2 and tune_tests folder. (#56799)
Migrate xgboost/lgbm benchmarks to train V2. (#56792)

Ray Tune

🎉 New Features:

Trigger Checkpointing via Trial / Tuner Callback. (#55527)

💫 Enhancements:

Improve _PBTTrialState for dev/debugging usage. (#56890)
Enable Train V2 in Tune unit tests and examples. (#56816)
Enable Train v2 in doc examples. (#56820)
Reintroduce keras tune callback. (#57121)

🔨 Fixes:

Increase tune checkpoint test latency threshold. (#56251)
Remove a bunch of low-signal/redundant train/air/tune tests. (#56477)
Remove tune_air_oom test. (#57089)

Ray Serve

🎉 New Features:

Add tests and DLQ business logic for async inference. (#55608)
Foundation work for aggregating metrics on controller. (#55568)
Include custom metrics method and report to controller. (#56005)
Add post scaling api. (#56135)
Introduce deployment rank manager. (#55729)
Integrated deployment ranks with deployment state. (#55829)
Add rank and world size in replica context. (#55827)
Added ssl to ray serve. (#55228)
Custom parameter for downscaling to zero. (#56573)
Add optional APIType filter to /api/serve/applications/ endpoint. (#56458)
Make deployment retry configurable. (#56530)

💫 Enhancements:

Aggregate autoscaling metrics on controller. (#56306)
Update metrics_utils for future global metrics aggregation in controller. (#55568)
Use deployment method in access logs for replicas. (#56829)
Cache router metrics. (#55897)
Allow same event loop handle shutdown from sync context. (#55551)
Additional deps to start with prometheus. (#57155)
Require prefix RAY_SERVE_ for env vars + value verification. (#55864)
Record queued metrics on timeseries. (#57024)
Add throughput opt env var for serve. (#55804)
Fix None pending Request. (#54775)
Omit unnecessary newlines in the config generated by serve build app:app. (#56609)
Expose actor name for target group api. (#56738)

🔨 Fixes:

Fix proxy lua dependency in dockerfile. (#57221)
Fix non thread safe asyncio task creation in router. (#56124)
Fix throughput optimized benchmarks. (#56173)
Move ingress validation for multiple fastapi deployment into client. (#56706)
Explicitly close choose_replicas_with_backoff async generator. (#56357)
Fix buffered logging reusing request context. (#56094)
Use default gc frequency for proxy. (#56511)
Fixing deployment scoped custom autoscaling. (#56192)

📖 Documentation:

Stable links for Ray serve. (#56241)
Add document for using fastapi factory pattern in serve. (#56607)
Add documentation for async inference (#56453)

🏗 Architecture refactoring:

Add microbenchmark for throughput optimized configuration. (#55900)
Only checkpoint controller state when it is confirmed that target state has changed (#55848)
Proxy Actor Interface. (#56288)
Allow ProxyActor to return true/false for health check. (#56660)

Ray Serve/Data LLM

🎉 New Features:

Score API Integration for Serve LLM. (#55914)
Add start/stop_profile method to LLMServer. (#55920)
Add prefix cache hit rate to Serve LLM dashboard. (#55675)
Configure aggregation interval for dashboard. (#56591)

💫 Enhancements:

Bump vLLM to 0.10.2. (#56535)
Vllm bump -> 0.10.1.1. (#56099)
Refactor: Improve Deployment Builder Ergonomics and Code Organization. (#57181)
Fix build_llm_processor for ServeDeploymentProcessor. (#57061)
Allow setting data_parallel_size=1 in engine_kwargs. (#55750)
Allow tuple for concurrency arg. (#55867)
Fix multimodal image extraction when no system prompt is present. (#56435)
Support azure and abfss in LLM config. (#56441)
Support custom s3 endpoint when downloading models from remote. (#55458)
Skip safetensor file downloads for runai streamer mode. (#55662)
Support colocating local DP ranks in DPRankAssigner. (#55720)
Adjust LLM engine timing logic. (#55595)
Fixed DP DSV3 issues. (#55802)
Gracefully return timeouts as HTTPException. (#56264)
Remove upstreamed workarounds 1/3. (#54512)

🔨 Fixes:

Changed LMCache dependency to use 0.3.3 to avoid regressions in the release test. (#56104)
Fix doc test for Working with LLMs guide. (#55917)
Fix sglang byod on release. (#55885)

📖 Documentation:

Add gpt oss deployment example. (#56400)
Add serve llm example to index page + other minor fix. (#56788)
Example serve llm deployment. (#55819)
Fix serve llm examples. (#56382)
Docs: serve llm deployment examples refinement. (#56287)
Add example of serving a VLLM model on fractional gpu. (#57197)
Add main pytest code snippet to those tests that were missing it. (#57167)

RLlib

🎉 New Features:

Add StepFailedRecreateEnv exception. (#55146)

💫 Enhancements:

Add tags to envrunner calls, count in flight requests in ActorManager. (#56930, #56953)
Add spaces in case only offline data is used. (#56141)
Add Footsies environment and tests. (#55041)

🔨 Fixes:

Fix failing env step in MultiAgentEnvRunner. (#55567)
Fix Metrics/Stats lifetime count and throughput measurement for async remote actors. (#56047)
Fixes Implementation of Shared Encoder. (#54571)
Fix MetricsLogger/Stats throughput bugs. (#55696)

📖 Documentation:

[RLlib] [DOC] Fix documentation typos and grammatical issues in RLlib docs (#56130)
Update rllib-env.rst - typo. (#56140)
Fixing typo in the RLlib documentation. (#55752)
Fix formatting of class references. (#55764)

🏗 Architecture refactoring:

Remove checkpoint release tests. (#57105)
Remove long_running_apex test. (#57097)
LINT: Enable ruff imports for multiple directories in rllib. (#56736)
Upgrade g3 to g4 machine for aws release test. (#56248)

Ray Core

🎉 New Features:

Alpha release of Ray Direct Transport
- Support ray.put() and ray.get() with nixl in gpu objects. (#56146)
- Support using ray.get with nixl to retrieve data from GPU object refs created by remote tasks. (#56559)
- Support tensor transfer from outside owners of actors. (#56485)
- Automatically enable tensor transport for the actor if any method specifies one. (#55324)
- Support cpu tensor transfer with NIXL in GPU Objects. (#55793)
- Handle multiple transfers of the same object to an actor. (#55628)
- Support NIXL as tensor transport backend. (#54459)
- Add a user-facing call to wait for tensor to be freed. (#55076)
- Always write to GPUObjectStore to avoid _get_tensor_meta() from hanging indefinitely. (#55433)
- Add warning when GPU object refs passed back to the same actor. (#55639)
- Avoid triggering a KeyError by the GPU object GC callback for intra-actor communication. (#54556)
Enable autoscaler v2 on clusters launched by the cluster launcher. (#55865)
Ray Symmetric Run Script and ray symmetric-run command. (#55111, #56497)

💫 Enhancements:

Ray Event Export
- GCS AddEvent support. (#55528)
- Actor event: add proto schema. (#56221)
- Node event: add proto schema and send node events to the aggregator. (#56031, #56426)
- Job event: add schema for driver job event and send job events to the aggregator. (#55032, #55213)
- Emit actor events to Event aggregator. (#56617)
- Export node event by default. (#56810)
Security
- Bind ray internal servers to the specified node ip instead of 0.0.0.0. (#55178, #55210, #55298, #55484)
- Bind dashboard agent http server to localhost in addition to the node ip. (#55910)
- Bind dashboard agent grpc to specified ip instead of 0.0.0.0. (#55732)
- Bind runtime env agent and dashboard agent http server to specified ip instead of 0.0.0.0. (#55431)
RPC network fault tolerance
- Making core worker pub sub RPCs fault tolerant. (#56436)
- Make RequestWorkerLease RPC Fault Tolerant. (#56191)
- Making ReturnWorkerLease Idempotent. (#56073)
- Making CancelWorkerLease RPC Fault Tolerant. (#56195)
- Make Free Objects RPC Fault Tolerant. (#56293)
- Make PinObjectIDs RPC Fault Tolerant. (#56443)
- Make Unsubscribe Idempotent. (#57546)
- Core Worker GetObjStatus GRPC Fault Tolerance. (#54567)
Not overriding accelerator id env vars when num_accelerators is 0 or not set. (#54928)
Migrate metric collection from opencensus to opentelemetry. (#53098, #53740)
Add per worker process group and deprecate process subreaper in favor of cleanup using process group. (#56476)
Add node_id validation in NodeAffinitySchedulingStrategy. (#56708)
Add io_context metrics to gcs and raylet. (#55762)
Modify RedisDelKeyPrefixSync to use the Redis SCAN command instead of KEYS. (#56907)
Add error_type to job failures. (#55578)
Add PID to structured logs for tasks and actors. (#55176)
Log actor name when warning about excess queueing. (#57124)
Output the error log on the driver side if the failed task will still retry. (#56472)
Prometheus http service discovery API. (#55656)
Add node ip in runtime env error message to improve debug observability. (#56837)
Fallback unserializable exceptions to their string representation. (#55476)
Introduce new exception type for un-pickleable exceptions. (#55878)
Improve docs for custom serialization for exceptions + add test. (#56156)
Add a warning when returning an object w/ num_returns=0. (#56213)
Adding ability to specify availability zones for ray cluster node pools on Azure. (#55532)
Query for supported Microsoft.Network/virtualNetworks API versions instead of relying on resource_client.DEFAULT_API_VERSION. (#54874)
Loosen Ray self-dependency check to allow matching versions. (#57019)
Add support for pip_install_options for pip. (#53551)
Proper typing for ObjectRef. (#55566)

🔨 Fixes:

Use subscription id from azure profile if not provided in config during AzureNodeProvider init. (#56640)
Always create standard public IP addresses (basic sku is deprecated). (#57131)
Fix: bug with config key pairs when launching worker nodes. (#57107)
If azure cluster launcher keypair doesnt exist create one automatically + doc typo fix. (#54596)
Fix "objects_valid" for the case that multiple instances of the same task are storing returns. (#54904)
Fix objects_valid check failure with except from BaseException. (#55602)
Preserve err type in case of task cancellation due to actor death. (#57538)
Fix checking for uv existence during ray_runtime setup. (#54141)
Prevent sending SIGTERM after calling Worker::MarkDead. (#54377)
Fixed the bug where the head was unable to submit tasks after redis is turned on. (#54267)
Fix possible race by checking node cache status instead of just subscription. (#54745)
Fix get actor timeout multiplier. (#54525)
Use a temporary file to share default worker path in runtime env. (#53653)
Fix check fail when task buffer periodical runner runs before RayEvent is initialized. (#55249)
Patch grpc with RAY_num_grpc_threads to control grpc thread count. (#54988)
Fix HandleRefRemoved thread safety. (#56445)
Fix error handling for plasma put errors. (#56070)
Fix batching logic in CoreWorkerPlasmaStoreProvider::Get. (#56041)
Fix RAY_CHECK failure during shutdown due to plasma store race condition. (#55367)
Fix autoscaler RAY_CHECK when GcsAutoscalerStateManager is out of sync with NodeManager. (#57010)
Fix bug where inflight requests are not taken into account by retryable. (#57142)
A timeout should be set when submitting patch requests for autoscaler. (#56605)
Fix the bug in memray regarding the default configuration of -o {output_file_path}. (#56732)
Fixed the issue of RemoveActorNameFromRegistry being called repeatedly. (#54955)
Fixed an issue where the command executed when use_podman=true and run_env=None was not prefixed with podman exec. (#56619)
Fix data race when using async gpu to gpu transfer. (#57112)
Retry + Make FreeActorObject idempotent. (#56447)
Fix check crash on gpu obj free if driver knows actor is dead. (#56404)
Handle system errors with a background monitor thread. (#56513)
Fix GPU metrics. (#56009)
Don't disconnect worker client on OBOD unless the worker is dead. (#57185)
Prevent stale GET request being registered if its lease was cleared. (#56766)
Drop messages received after ClientConnection::Close. (#56240)
Fix cancel race that leads to RAY_CHECK it->second.submitted_task_ref_count > 0. (#56123)
Reorder asyncio actor shutdown to terminate asyncio thread first. (#56827)
Fix actor import error message for async actors. (#55722)
Allow task manager access with submitter mutex + unify retry. (#56216)
Fix bug in restore_from_path such that connector states are also restored on remote EnvRunners. (#54672)
Fix S3 access issue in AKS. (#56358)
Add S3 public bucket fallback to handle NoCredentialsError. (#56334)
Fix ABFSS (Azure Blob File System Secure) protocol support problems during E2E test. (#56188)
Ray cluster commands (up, attach, status, etc) updates to work on Windows. (#54982)
Update cluster scheduler to handle label selector hard node id constraint. (#56235)

📖 Documentation:

Added guide on using type hints with Ray Core. (#55013)
Lifecycle of a task. (#55496)
Add OSS Document for Task Events. (#56203)
Fix Missing Events Issue in Task Events. (#55916)
Add docs for asyncio and object mutability. (#56790)
Update getting started and set up document for ray on vsphere. (#56954)
Docfix - rst annotation showing up in render. (#57104)
Add threading requirement to NodeProvider interface. (#56349)
Add guidance for matching Ray and Python versions with uv envs. (#56597)
Fix documentation typos, grammar, and terminology inconsistencies. (#56066, #56067, #56068, #56069, #56128, #56129, #56130, #56131, #56132, #56272, #56273, #56274, #56275, #56277, #56278, #56279)
Update SLURM docs with symmetric-run. (#56775)
Update Kueue integration documentation to include RayService & RayCluster support. (#56781)
Application Gateway for Containers as ingress to access Ray Cluster. (#56574)
Update DLAMI Information in aws.md. (#55702)

Dashboard

💫 Enhancements:

Use pynvml for GPU metrics. (#56000)
Default dashboard usability improvements. (#55620)
Make Ray Train Dashboard Panel Ids Static. (#55559)
Small fixes to Metrics Tab for kube-ray clusters. (#57149)
Add metadata to indicate full dashboard embedding is supported. (#56077)
Use ray node id instead of ip for profilinglink. (#55439)
Fix grafana dashboard generation bug. (#56346)
Catch OSError when detecting the GPU. (#56158)

🔨 Fixes:

Removed references to a deleted Data metrics panel. (#55478)
Fix typo in memory_utils and adjust display formatting for clarity. (#56217)

Ray Images

🎉 New Features:

Add support for building and publishing ray-extra images. (#56543)
Add ray-llm and ray-ml extra images. (#56800)
Build ray-extra images for aarch64. (#56818)
Add slim image to the image build matrix. (#55723)

💫 Enhancements:

Add haproxy binary, for ray serve use. (#56845)
Add ~/.local/bin to PATH in slim image. (#56920)
Remove slim's dependency on normal bases. (#56544)
Add label for ray version and commit. (#56493)
Refactor apt package installation. (#55701)
Allow using explicit base type. (#56545)
Add extra-test stage in image building. (#55725)
Add test rules for image building files. (#56554)
Add ray-llm image type check. (#56542)
Unify label and tag conventions. (#56189)
GKE GPU compat paths: PATH, LD_LIBRARY_PATH (temporarily). (#55569)
Stop publishing ray-ml images. (#57070)
Stop building and releasing x86 osx wheels. (#57077)

📖 Documentation:

Update latest Docker dependencies for 2.49.0 release. (#55966)
Update latest Docker dependencies for 2.49.2 release. (#56760)

Wheels and images

💫 Enhancements:

Use bazel run to generate files required for the wheel and testing (#55957, #56527, #56969, #56004, #55928)
Ban click 8.3.0. (#56789)
Upgrade protobuf to v4. (#54496)
Add adlfs[abfs] into image (#56084)
Upgrade boto3 to 1.29.x. (#56363)
Upgrading orjson to 3.9.15. (#55972)
Update spdlog to 15.3. (#56711)

Thanks!

Thank you to everyone who contributed to this release!

@alexeykudinkin, @richardliaw, @nrghosh, @ljstrnadiii, @Daraan, @kouroshHakha, @Bye-legumes, @kamil-kaczmarek, @jugalshah291, @sampan-s-nayak, @jjyao, @Evelynn-V, @gangsf, @omatthew98, @TimothySeah, @kshanmol, @goutamvenkat-anyscale, @axreldable, @jiangwu300, @simonsays1980, @400Ping, @JasonLi1909, @chuang0221, @weiliango, @Myasuka, @win5923, @liulehui, @khluu, @ok-scale, @eicherseiji, @tianyi-ge, @MengjinYan, @kevin85421, @Yevet, @orangeQWJ, @vie-serendipity, @edoakes, @wyhong3103, @israbbani, @vickytsang, @HassamSheikh, @acrewdson, @czgdp1807, @daiping8, @carolynwang, @thc1006, @jeffreyjeffreywang, @Stack-Attack, @Catch-Bull, @elliot-barn, @Levi080513, @BestVIncent, @dragongu, @jmajety-dev, @jcarlson212, @tohtana, @abrarsheikh, @crypdick, @Yicheng-Lu-llll, @ZacAttack, @justinvyu, @lk-chen, @alanwguo, @mcoder6425, @my-vegetable-has-exploded, @yancanmao, @arcyleung, @rjpower, @codope, @harshit-anyscale, @dayshah, @stephanie-wang, @KaisennHu, @ryanaoleary, @saihaj, @mattip, @rueian, @Kunchd, @pavitrabhalla, @owenowenisme, @Aydin-ab, @gvspraveen, @minerharry, @JackGammack, @jpatra72, @coqian, @zcin, @dstrodtman, @aslonnie, @ahao-anyscale, @GuyStone, @iamjustinhsu, @seanlaii, @ruisearch42, @akyang-anyscale, @ArturNiederfahrenhorst, @bveeramani, @OneSizeFitsQuorum, @xinyuangui2, @sb-hakunamatata, @22quinn, @Sparks0219, @sven1977, @snehachhabria, @dioptre, @nadongjun, @eric-higgins-ai, @marosset, @MatthewCWeston, @pcmoritz, @can-anyscale, @pimdh, @roshankathawate, @matthewdeng, @martinbomio, @GokuMohandas, @alimaazamat, @ali-corpo, @landscapepainter, @Qiaolin-Yu, @vaishdho1, @avigyabb, @srinathk10, @tannerdwood

Ray-2.49.2

2025-09-19T18:10:36Z

There is no difference between 2.49.2 and 2.49.1, though we needed a patch version for other out of band reasons. To fill the awkward blankness, here is a haiku about Ray:

Summit drawing near
Ray advances, step by step
Scaling without end