Studio: Fix export subprocess crash by restoring transformers activation helper#4959
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces the activate_transformers_for_subprocess function to manage transformers versions for worker subprocesses by updating sys.path and PYTHONPATH. A critical concern was raised regarding the call to _resolve_base_model, which might trigger a premature import of the transformers library, effectively locking the process into the default version and rendering the subsequent environment switch ineffective.
| prepends ``.venv_t5`` to ``sys.path`` and propagates that path via | ||
| ``PYTHONPATH`` so child processes (e.g. GGUF converter) inherit it. | ||
| """ | ||
| resolved = _resolve_base_model(model_name) |
There was a problem hiding this comment.
Calling _resolve_base_model here carries a risk of prematurely loading the transformers library before the environment is switched. Specifically, _resolve_base_model (at line 118) can lazily import utils.models, which is listed in _PURGE_PREFIXES as a module that holds references to transformers internals. If this import triggers a transformers load, the subprocess will be locked into the default version, and subsequent attempts to switch versions via sys.path will be ignored because the module is already in sys.modules.
Since this function is intended to be called before any ML imports, consider if _resolve_base_model can be made safe (e.g., by avoiding the heavy fallback that imports utils.models) or if the path switch should happen earlier.
|
This is no longer needed as rebasing from main branch into |
Issue:
Exporting unsloth/qwen2.5-vl-3b-instruct-unsloth-bnb-4bit checkpoint failed with
Failed to activate transformers version: cannot import name 'activate_transformers_for_subprocess' ...Fixes GGUF/export subprocess startup failure caused by a missing
activate_transformers_for_subprocesssymbol.Restores a backward-compatible helper in
backend/utils/transformers_version.pysocore/export/worker.pycan activate the correct transformers environment before ML imports.Root Cause
core/export/worker.pyimportedactivate_transformers_for_subprocess, but the function had been removed fromutils/transformers_version.pyin this branch, causing import-time export failure.Test plan