Security Digest · Mar 26, 2024 · Misconfiguration · AI / LLM

ShadowRay: Exposed Ray Clusters Hijacked (CVE-2023-48022)

Ray's Jobs API runs any code without auth by design; Oligo found exposed AI clusters hijacked since Sept 2023. Disputed CVE, no patch: firewall it, use tokens.

Our verdict

Check exposure

There is no patch to install: if any Ray dashboard, Jobs or Client port is reachable from an untrusted network, anyone who can reach it can run code on your cluster.

Who must act: Anyone running Ray clusters (self-managed, on Kubernetes/KubeRay or on cloud VMs) whose head node ports, especially the dashboard/Jobs port 8265, are reachable from the internet or other untrusted networks.

Who can relax: Teams whose Ray clusters are only reachable from a private, controlled network or through an authenticating proxy or SSH tunnel, and managed-service users whose provider handles isolation.

Affected
Ray clusters with the dashboard/Jobs API reachable from untrusted networks (the CVE record names Ray 2.6.3 and 2.8.0, but the behavior is by design across versions)
Fixed in
No fix for CVE-2023-48022 (disputed)
Optional built-in token authentication from Ray 2.52.0 (RAY_AUTH_MODE=token)
Four related Ray CVEs (CVE-2023-6019, -6020, -6021, -48023) fixed in Ray 2.8.1
CVSS
9.8 (v3.1, CISA ADP)
EPSS
83.9% (percentile 99.7)
CISA KEV
Not listed
Exploitation
Exploited in the wild
Severity
Critical

What to do

  1. Inventory Ray head nodes and confirm that ports such as 8265 are not reachable from the internet (security groups, firewall rules, Kubernetes Services).
  2. Do not bind Ray to 0.0.0.0 on public interfaces; use a private interface or network.
  3. If remote access is needed, put the dashboard behind an authenticating proxy, VPN or SSH tunnel.
  4. Upgrade to Ray 2.52.0 or later and enable token authentication (RAY_AUTH_MODE=token, ray get-auth-token --generate) as defense in depth.
  5. Upgrade past Ray 2.8.1 to pick up fixes for the four related CVEs.
  6. If a cluster was exposed, rotate every credential it could reach (cloud keys, OpenAI/Hugging Face tokens, database and SSH keys).

How to check if you are affected or compromised

  • Test from outside your network whether http://<head-node>:8265 responds; if it does, assume it may have been used.
  • Review Ray job history (ray job list, or the dashboard Jobs tab) for jobs you did not submit.
  • Look for cryptominer processes (XMRig, NBMiner, Zephyr miners), unexpected GPU/CPU usage, reverse shells, or traffic to interactsh domains on cluster nodes.

What happened#

On March 26, 2024, Oligo Security reported an active campaign, which it named ShadowRay, against internet-exposed clusters running Ray. Ray is the open-source framework from Anyscale that many teams use to scale AI training and inference. Attackers were using Ray's Jobs API, which accepts and runs code from any caller with no authentication. Oligo dated the earliest evidence to September 5, 2023, which means some clusters were compromised for at least seven months before the report.

The flaw is tracked as CVE-2023-48022, published in November 2023. It was one of five Ray issues that Bishop Fox, Sierra Haex and Protect AI reported to Anyscale in late 2023. The record is tagged disputed. It describes remote code execution via the job submission API in Ray 2.6.3 and 2.8.0, and quotes the vendor's position that Ray "is not intended for use outside of a strictly controlled network environment." CISA's ADP enrichment on the record scores it CVSS 3.1 9.8. Anyscale's November 30, 2023 post said the lack of built-in authentication was a long-standing design decision rather than a bug. The same post said four related CVEs (CVE-2023-6019, -6020, -6021 and -48023) were fixed for Ray 2.8.1.

Oligo said the compromised machines exposed OpenAI, Stripe and Slack credentials, Hugging Face tokens, SSH keys, password hashes, cloud credentials (often with root access) and Kubernetes API access. Attackers installed cryptominers (XMRig, NBMiner and Java-based Zephyr miners) and reverse shells. Oligo's own estimate put the value of the hijacked compute at "almost a billion USD." That figure is a projection from cloud on-demand pricing, not a measured loss.

Who is affected#

Any Ray cluster whose dashboard, Jobs or Client port can be reached from an untrusted network. The Ray security documentation says that anyone who can reach those ports can execute arbitrary code on the cluster. This is not limited to the two versions named in the CVE record. It is how Ray works unless you put controls in front of it. The CISA KEV catalog does not list CVE-2023-48022 as of October 21, 2026, even though exploitation is documented.

Do you really need to act?#

That depends on network exposure. If your clusters sit in a private VPC and are reached only through a VPN, SSH tunnel or authenticating proxy, this campaign did not affect you, and no upgrade is needed to close a hole. If any head node was ever reachable from the internet, treat it as a probable compromise. Oligo's data shows that attackers were scanning and exploiting these clusters for months, and GPU clusters often hold valuable credentials.

The picture has improved. The Ray docs say built-in token authentication arrived in Ray 2.52.0 as an opt-in feature. The same docs say it becomes the default for local clusters in 2.59 and for all clusters in 2.61. Ray is explicit that tokens are "not an alternative" to network isolation. Tokens travel as plaintext HTTP headers unless you add TLS, and they do not expire.

What to do now#

  1. List every Ray head node and test from outside your network whether port 8265 (the dashboard and Jobs API) or other Ray ports answer.
  2. Close public exposure with security groups, firewall rules or internal-only Kubernetes Services. Avoid binding to 0.0.0.0 on public interfaces.
  3. For remote users, use an authenticating proxy, a VPN, or the ray dashboard cluster.yaml port-forwarding command.
  4. Upgrade to Ray 2.52.0 or later and enable tokens: ray get-auth-token --generate, then RAY_AUTH_MODE=token ray start --head, with the same token on every node.
  5. If a cluster was exposed, rotate cloud keys, model-hub tokens, database passwords and SSH keys it could reach, and rebuild its nodes.

How to tell if you were hit#

  • Run ray job list or check the dashboard's Jobs view for jobs your team did not submit.
  • Look for miner processes, unexplained GPU saturation, reverse shells, or DNS lookups for interactsh domains, which Oligo saw attackers use to evade detection.
  • Check cloud audit logs for API calls made with credentials that were present on cluster nodes.

Timeline#

  • — Earliest exploitation evidence Oligo found.
  • — CVE-2023-48022 published.
  • — Anyscale disputes the CVE and says four related fixes will ship in Ray 2.8.1.
  • — Oligo discloses the ShadowRay campaign.
  • — Token authentication available since Ray 2.52.0. The CVE remains disputed and is not in KEV.

Vulnerability records

  • CVE-2023-48022 · Anyscale Ray 2.6.3 and 2.8.0 allows a remote attacker to execute arbitrary code via the job submission API.

Sources

Original sources first: vendor and project advisories, CVE and government records, the discoverers’ write-ups. Every claim above links to one of these. See how the Security Desk works.

Original and primary sources

  1. CVE.org — CVE-2023-48022 cve.org
  2. Oligo Security — ShadowRay: First Known Attack Campaign Targeting AI Workloads Actively Exploited In The Wild oligo.security
  3. Anyscale — Update on Ray CVEs anyscale.com
  4. Ray documentation — Security docs.ray.io
  5. Ray documentation — Token authentication docs.ray.io
  6. CISA — Known Exploited Vulnerabilities Catalog cisa.gov