Interactive-LLM-Powered-NPCs
Interactive LLM Powered NPCs, is an open-source project that completely transforms your interaction with non-player characters (NPCs) in any game! 🎮🤖🚀
What is Interactive-LLM-Powered-NPCs?
Interactive LLM Powered NPCs, is an open-source project that completely transforms your interaction with non-player characters (NPCs) in any game! 🎮🤖🚀
Overview of Interactive-LLM-Powered-NPCs
What it is
Interactive LLM Powered NPCs is an open-source project that transforms interaction with non-player characters (NPCs) in games. It allows players to speak into a microphone and have conversations with NPCs in games such as Cyberpunk 2077, Assassin's Creed, and GTA 5. The project uses sadtalker for lip synchronization, facial recognition to identify characters, vector stores for NPC memory, and pre-conversation files to shape dialogue style. It also captures facial expressions via webcam to adjust NPC responses. The system works without modifying game source code by replacing facial pixels generated by the game.
Key features
- Microphone input: voice converted to text for NPC interaction.
- Facial recognition: identifies which NPC the player is engaging with.
- LLM integration: uses a Large Language Model to generate context-aware responses.
- Pre-conversation files: iconic lines and style examples passed to the LLM.
- Facial animation and speech generation: text-to-speech and sadtalker produce animated video.
- Emotion recognition: webcam captures player's facial emotions to influence NPC responses.
- Non-visual interactions: players can speak an NPC's name to start conversation even when face is not visible.
- Game compatibility: works with any game without modifying source code.
Use cases
- Enhancing dialogue in open-world games like Cyberpunk 2077, Assassin's Creed Valhalla, and GTA 5.
- Allowing players to have unscripted conversations with background and side characters.
- Creating immersive role-playing experiences where NPCs remember past interactions.
- Adding emotional depth to NPC interactions through facial expression recognition.
Getting started
sh git clone https://github.com/AkshitIreddy/Interactive-LLM-Powered-NPCs.git cd Interactive-LLM-Powered-NPCs python -m venv .venv .venv/scripts/activate pip install -r requirements.txt cd sadtalker bash scripts/download_models.sh Additional prerequisites include Python 3.10.6, GIT, wget, Microsoft Build Tools and Visual Studio, and FFmpeg. After installation, create a Cohere account (free) and add the API key to apikeys.json. Delete all files in video_temp and temp folders.
License and pricing
The project is licensed under the MIT License. It is open-source and free to use.
Based on the project's public README and repository metadata.