Add Support for Local LLama Models via LlamaCpp Integration #781
Conversation
![sweep-ai[bot]](https://avatars.githubusercontent.com/in/307814?s=60&v=4){kind=small}
|
No CI runs were found. Sweep can fix CI failures, so we recommend setting it up for your repository to let Sweep run your build and tests. If you think this is an error, check Sweep's permissions at https://github.com/organizations/kyaukyuai/settings/installations/45715441. 📖 For more information on how to use Sweep, please read our documentation. Tracking ID: 27853a050e-ci-0
|
|
Walkthroughこの変更は、LLamaモデルのローカルサポートを追加するためのものです。新たな環境変数の追加、LLamaモデルの作成と初期化のための関数の更新、そして新たなユニットテストの追加が含まれています。これらの変更は、コードの外部インターフェースや振る舞いに影響を与える可能性があります。 Changes
Uplevel your code reviews with CodeRabbit ProCodeRabbit ProIf you like this project, please support us by purchasing the Pro version. The Pro version has advanced context, superior noise reduction and several proprietary improvements compared to the open source version. Moreover, CodeRabbit Pro is free for open source projects. |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Files selected (3)
- .env.sample (2)
- gpt_all_star/core/llm.py (3)
- tests/core/test_llm.py (3)
Review comments generated (8)
- Review: 5
- LGTM: 3
Tips
Chat with 
CodeRabbit Bot (@coderabbitai)
- Reply on review comments left by this bot to ask follow-up questions. A review comment is a comment on a diff or a file.
- Invite the bot into a review comment chain by tagging
@coderabbitaiin a reply.
Code suggestions
- The bot may make code suggestions, but please review them carefully before committing since the line number ranges may be misaligned.
- You can edit the comment made by the bot and manually tweak the suggestion if it is slightly off.
Pausing incremental reviews
- Add
@coderabbitai: ignoreanywhere in the PR description to pause further reviews from the bot.
| from langchain_community.llms import LlamaCpp | ||
| from langchain.callbacks.manager import CallbackManager | ||
| from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler | ||
|
|
||
|
|
||
| class LLM_TYPE(str, Enum): | ||
| OPENAI = "OPENAI" | ||
| AZURE = "AZURE" | ||
| ANTHROPIC = "ANTHROPIC" | ||
| LLAMA = "LLAMA" |
There was a problem hiding this comment.
新たにLlamaCppのインポートとLLM_TYPE列挙型にLLAMAオプションを追加しています。これにより、LLamaモデルのサポートが追加されます。
| elif llm_name == LLM_TYPE.LLAMA: | ||
| return _create_chat_llama( | ||
| model_path=os.getenv("LLAMA_MODEL_PATH"), | ||
| temperature=0.1, | ||
| n_ctx=int(os.getenv("LLAMA_N_CTX", "2048")), | ||
| n_gpu_layers=int(os.getenv("LLAMA_N_GPU_LAYERS", "0")), | ||
| ) |
There was a problem hiding this comment.
create_llm()関数が更新され、LLM_TYPE.LLAMAの場合に_create_chat_llama()関数を呼び出すようになりました。これにより、LLamaモデルの作成が可能になります。
| def _create_chat_llama( | ||
| model_path: str, | ||
| temperature: float, | ||
| n_ctx: int = 2048, | ||
| n_gpu_layers: int = 0, | ||
| ) -> LlamaCpp: | ||
| callback_manager = CallbackManager([StreamingStdOutCallbackHandler()]) | ||
| return LlamaCpp( | ||
| model_path=model_path, | ||
| temperature=temperature, | ||
| n_ctx=n_ctx, | ||
| n_gpu_layers=n_gpu_layers, | ||
| callback_manager=callback_manager, | ||
| verbose=True, | ||
| ) |
There was a problem hiding this comment.
新たに_create_chat_llama()関数が作成されました。この関数は、LLamaモデルの初期化を行い、LlamaCppインスタンスを返します。
| @pytest.fixture | ||
| def mock_llamacpp(): | ||
| with patch("gpt_all_star.core.llm.LlamaCpp") as mock: | ||
| yield mock | ||
|
|
||
|
|
||
| @pytest.fixture | ||
| def mock_callback_manager(): | ||
| with patch("gpt_all_star.core.llm.CallbackManager") as mock: | ||
| yield mock |
There was a problem hiding this comment.
新たにmock_llamacppとmock_callback_managerという2つのfixtureが追加されています。これらは、LlamaCppとCallbackManagerのモックを作成し、テストで使用します。問題はありません。
| def test_create_chat_llama(mock_llamacpp, mock_callback_manager): | ||
| model_path = "/path/to/model.gguf" | ||
| temperature = 0.1 | ||
| n_ctx = 2048 | ||
| n_gpu_layers = 0 | ||
|
|
||
| _create_chat_llama( | ||
| model_path=model_path, | ||
| temperature=temperature, | ||
| n_ctx=n_ctx, | ||
| n_gpu_layers=n_gpu_layers, | ||
| ) | ||
|
|
||
| mock_callback_manager.assert_called_once() | ||
| mock_llamacpp.assert_called_once_with( | ||
| model_path=model_path, | ||
| temperature=temperature, | ||
| n_ctx=n_ctx, | ||
| n_gpu_layers=n_gpu_layers, | ||
| callback_manager=mock_callback_manager.return_value, | ||
| verbose=True, | ||
| ) |
There was a problem hiding this comment.
新たにtest_create_chat_llamaというテスト関数が追加されています。この関数では、_create_chat_llama関数が期待通りに動作するかを検証しています。具体的には、LlamaCppとCallbackManagerが適切に呼び出され、正しいパラメータで初期化されることを確認しています。問題はありません。
Purpose
Extend the GPT All-Star library to support local LLama models using LangChain's LlamaCpp integration, enabling users to run large language models locally without relying on cloud-based API services.
This pull request was created to solve the following GitHub issue:
Add llama3.3 support
Details
Let's add opensource llama model. can you give me a hint so I can try it out
Add Llama 2 LLM Support to Core LLM Module
Description:
Extend the LLM module to support Llama 2 models through LangChain's integration, following the existing pattern for other LLM providers.
Tasks:
In
gpt_all_star/core/llm.py:LLAMAtoLLM_TYPEenum_create_chat_llamahelper functioncreate_llmto handle Llama caseIn
.env.sample:Test:
tests/core/test_llm.py:Implementation Notes:
Description
This pull request introduces comprehensive support for local LLama models by:
LLAMAoption to theLLM_TYPEenum_create_chat_llama()function to initialize LlamaCpp modelscreate_llm()function to handle Llama model creation.env.samplefile to include Llama-specific configuration parametersThe implementation allows flexible configuration of Llama models through environment variables, including model path, context size, and GPU layer configuration. This provides users with a seamless way to integrate local, open-source language models into their existing workflows.
Summary
LLM_TYPE.LLAMAenum option_create_chat_llama()functioncreate_llm()to support Llama models.env.samplewith Llama configurationLlamaCppfrom LangChainFixes
Fixes #780. Continue the conversation here: https://app.sweep.dev/c/dca03bf8-dbba-467c-bdc0-8262932c6245.
To have Sweep make further changes, please add a comment to this PR starting with "Sweep:".
📖 For more information on how to use Sweep, please read our documentation.
LLAMAを選択した場合の設定が可能になります。