get_num_tokens_from_messages implicitly requires transformers

[g-pavlov]

Checked other resources

• I added a very descriptive title to this issue.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.
• I am sure that this is a bug in LangChain rather than my code.
• The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

Consider the following code

model = init_chat_model(
        "anthropic.claude-3-5-sonnet-20240620-v1:0",
        model_provider="aws_bedrock", 
        temperature=0
    )
model.get_num_tokens_from_messages(messages)

Error Message and Stack Trace (if applicable)

  File "sample.py", line 74, in sample_func
    token_count = model.get_num_tokens_from_messages(messages)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/langchain_core/language_models/base.py", line 393, in get_num_tokens_from_messages
    return sum(self.get_num_tokens(get_buffer_string([m])) for m in messages)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/langchain_core/language_models/base.py", line 393, in <genexpr>
    return sum(self.get_num_tokens(get_buffer_string([m])) for m in messages)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/langchain_core/language_models/base.py", line 366, in get_num_tokens
    return len(self.get_token_ids(text))
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/langchain_core/language_models/base.py", line 353, in get_token_ids
    return _get_token_ids_default_method(text)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/langchain_core/language_models/base.py", line 77, in _get_token_ids_default_method
    tokenizer = get_tokenizer()
                ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/langchain_core/language_models/base.py", line 69, in get_tokenizer
    raise ImportError(msg) from e
ImportError: Could not import transformers python package. This is needed in order to calculate get_token_ids. Please install it with `pip install transformers`.

Description

The code implicitly yields import of the transformers library.

Unfortunately this is not a defined dependency of langchain and neither it is documented as required and when.

Since this is yet another model function it makes sense to treat it a hard dependency and require it to be automatically installed with the langchain itself.
Or alternatively a lightweight fallback could be implemented with the documented option to install the full transformers library.

System Info

System Information

OS: Darwin
OS Version: Darwin Kernel Version 24.3.0: Thu Jan 2 20:24:23 PST 2025; root:xnu-11215.81.4~3/RELEASE_ARM64_T6020
Python Version: 3.12.4 (main, Jun 6 2024, 18:26:44) [Clang 15.0.0 (clang-1500.3.9.4)]

Package Information

langchain_core: 0.3.41
langchain: 0.3.20
langchain_community: 0.3.19
langsmith: 0.3.12
langchain_aws: 0.2.15
langchain_cli: 0.0.31
langchain_experimental: 0.3.4
langchain_huggingface: 0.1.2
langchain_ollama: 0.2.3
langchain_openai: 0.2.14
langchain_text_splitters: 0.3.6
langchain_together: 0.3.0
langgraph_sdk: 0.1.55
langserve: 0.3.1

Other Dependencies

aiohttp: 3.11.13
aiohttp<4.0.0,>=3.8.3: Installed. No version info available.
async-timeout<5.0.0,>=4.0.0;: Installed. No version info available.
boto3: 1.37.8
dataclasses-json<0.7,>=0.5.7: Installed. No version info available.
fastapi: 0.115.11
gitpython: 3.1.44
gritql: 0.1.5
httpx: 0.27.2
httpx-sse<1.0.0,>=0.4.0: Installed. No version info available.
huggingface-hub: 0.29.2
jsonpatch<2.0,>=1.33: Installed. No version info available.
langchain-anthropic;: Installed. No version info available.
langchain-aws;: Installed. No version info available.
langchain-cohere;: Installed. No version info available.
langchain-community;: Installed. No version info available.
langchain-core<1.0.0,>=0.3.34: Installed. No version info available.
langchain-core<1.0.0,>=0.3.41: Installed. No version info available.
langchain-deepseek;: Installed. No version info available.
langchain-fireworks;: Installed. No version info available.
langchain-google-genai;: Installed. No version info available.
langchain-google-vertexai;: Installed. No version info available.
langchain-groq;: Installed. No version info available.
langchain-huggingface;: Installed. No version info available.
langchain-mistralai;: Installed. No version info available.
langchain-ollama;: Installed. No version info available.
langchain-openai;: Installed. No version info available.
langchain-text-splitters<1.0.0,>=0.3.6: Installed. No version info available.
langchain-together;: Installed. No version info available.
langchain-xai;: Installed. No version info available.
langchain<1.0.0,>=0.3.20: Installed. No version info available.
langserve[all]: Installed. No version info available.
langsmith-pyo3: Installed. No version info available.
langsmith<0.4,>=0.1.125: Installed. No version info available.
langsmith<0.4,>=0.1.17: Installed. No version info available.
numpy: 2.2.3
numpy<3,>=1.26.2: Installed. No version info available.
ollama: 0.4.7
openai: 1.65.4
orjson: 3.10.15
packaging: 24.2
packaging<25,>=23.2: Installed. No version info available.
pydantic: 2.10.6
pydantic-settings<3.0.0,>=2.4.0: Installed. No version info available.
pydantic<3.0.0,>=2.5.2;: Installed. No version info available.
pydantic<3.0.0,>=2.7.4: Installed. No version info available.
pydantic<3.0.0,>=2.7.4;: Installed. No version info available.
pytest: 8.3.5
PyYAML>=5.3: Installed. No version info available.
requests: 2.32.3
requests-toolbelt: 1.0.0
requests<3,>=2: Installed. No version info available.
rich: 13.9.4
sentence-transformers: 3.4.1
SQLAlchemy<3,>=1.4: Installed. No version info available.
sse-starlette: 1.8.2
tenacity!=8.4.0,<10,>=8.1.0: Installed. No version info available.
tenacity!=8.4.0,<10.0.0,>=8.1.0: Installed. No version info available.
tiktoken: 0.7.0
tokenizers: 0.21.0
tomlkit: 0.12.5
typer[all]: Installed. No version info available.
typing-extensions>=4.7: Installed. No version info available.
uvicorn: 0.23.2
zstandard: 0.23.0