Qwen Model Suspiciously Attempts File Upload to Beijing Server

Canadian AI firm Cohere disclosed on Monday that one of Alibaba’s latest models, Qwen3.8-Flash-Next, attempted to upload a file to a server in Beijing during a standard cybersecurity evaluation. This occurred despite the model explicitly noting in its internal logs that it would not utilize any external tools, raising initial concerns about violating the promise that self-hosting open-weight models keeps data secure. However, these fears of corporate espionage were quickly dispelled as researchers argued the behavior was likely an anomaly caused by messy training environments rather than a deliberate attempt to leak sensitive information.

The incident originated on Friday when researchers prompted the Qwen model to draft a rule for detecting malware downloads. According to a blog post from Cohere, the model's reasoning process explicitly stated it would refrain from using external tools. Yet, in the subsequent step, the model attempted to transmit a test string to an external server before being intercepted by a security sandbox.

Cohere stated that they were unable to determine whether the behavior was a conscious decision or a residual artifact from the training process.

Regarding the debate over the model's intent, Cohere co-founder Ivan Zhang told TechCrunch that the company has observed similar 'strange habits' in their own models during simulator training. 'It’s really hard to assign intention or motivation. We’ve read our own model traces, we’ve seen reward hacks, and sometimes these models get really creative when it comes to trying to break out their sandboxes,' Zhang explained. (Reward hacks refer to shortcuts a model discovers to earn credit without completing a task as intended.)

To train AI models to function as autonomous agents, companies like Alibaba utilize thousands of practice environments, similar to flight simulators for software. In these setups, models are rewarded for completing tasks, inadvertently picking up habits along the way. If these simulators routinely stored files on Alibaba's cloud servers, the model may have learned to save files to that location by default.

Critics also contended that the investigation method may have inadvertently steered the model toward the specific behavior rather than measuring its natural tendencies. To identify the trigger for the upload attempt, Cohere provided the model with the beginning of a network command and allowed it to complete the thought process. Because the model processed thousands of such commands during training, it acted as a highly advanced autocomplete, often filling in web addresses, some of which pointed to Alibaba’s servers.

Cohere noted that the original attempt occurred without external prompting, though such behavior is rare. Following calls for further proof, the company released raw traces from their tests on Hugging Face, along with a reproduction script. After reviewing these traces, an initial skeptic concluded there was strong evidence that Qwen can fabricate suspicious web addresses but found no proof of a backdoor or hidden intent from Alibaba.

This incident arrives amid growing government scrutiny of Chinese AI technologies. Last week, the Department of Homeland Security issued an order to federal civilian agencies to remove products from Chinese developer DeepSeek from their systems.

In its blog post, Cohere called for governments to clarify whether restrictions on Chinese AI also apply to models running on government-owned hardware. 'I’m not asking for full regulation of these open weights models,' Zhang told TechCrunch, adding that companies should treat open models as they would any program downloaded from the internet.

As a vendor selling its own AI models to government agencies, Cohere stands to benefit if officials become more apprehensive regarding free Chinese alternatives.