LLM Engineer
Company Description
BumbleB is an enterprise software company that specializes in building software as a service (SaaS) and platform as a service (PaaS) solutions in the field of data insights and artificial intelligence (AI). Our goal is to enable businesses to leverage the power of AI and data to drive innovation and make intelligent decisions.
Role Description
This is a full-time role (location Bengaluru) for an LLM Engineer at BumbleB. You will own the behavior layer of our AI products — understanding how language models generate responses, how conversations are structured, and how to systematically tune model outputs for quality, safety, and reliability. You will work closely with the founding team to design prompt architectures, evaluate model performance, and build tooling around LLM integrations.
What You’ll Do
- Design and iterate on prompt strategies, system instructions, and conversation flows
- Evaluate and benchmark LLM outputs for accuracy, coherence, tone, and safety
- Build evaluation frameworks and automated testing pipelines for model behavior
- Implement retrieval-augmented generation (RAG) and grounding techniques
- Fine-tune and adapt models for domain-specific enterprise use cases
- Collaborate with engineering to integrate LLMs into production systems
Qualifications
- Strong programming skills in Python
- Deep understanding of how large language models work — tokenization, attention, decoding strategies, context windows
- Experience with prompt engineering, few-shot learning, and chain-of-thought techniques
- Hands-on experience with LLM APIs and frameworks (OpenAI, Anthropic, LangChain, LlamaIndex, or similar)
- Familiarity with model evaluation methods — both automated metrics and human evaluation
- Understanding of conversation design — multi-turn dialogue, context management, and guardrails
- Experience with fine-tuning, RLHF, or preference optimization is a plus
- Familiarity with vector databases and embedding models
- 5+ years of industry experience in software engineering or applied AI
- Ability to work independently
- Excellent problem-solving and analytical thinking skills
- Effective written and verbal communication skills
- Bachelor’s or Master’s degree in Computer Science, Computational Linguistics, AI, or a related field