Hari Prasath S

EXPERIENCE

Software Development Engineer

Co-developed an LLMOps platform layer on Agno and LiteLLM, separating agent orchestration from provider access and centralizing LLM APIs, LLM observability and security in one gateway.
Built a stateless tool server with FastMCP and FastAPI, enabling agent tool calling for user-authored Python tools by dynamically provisioning short-lived MCP servers — horizontally scalable without per-tool Lambdas.
Architected and built an AI governance layer for PII protection, de-identifying personal data before LLM calls and restoring it for tools and responses, preserving conversational coherence.
Built real-time voice agents on LiveKit Agents that join and converse in live meetings via STT, TTS, and voice activity detection, with speech usage metering.
Built an agentic NL2SQL analytics assistant with Google ADK and Gemini, translating natural-language questions into parameterized SQL and synthesizing results into actionable insights.

Built multimodal RAG for AI agents using embeddings and Qdrant to retrieve relevant context from user-provided data, grounding agent responses and tool use in that context.
Built LLM-as-a-Judge model evaluation for AI agents, benchmarking candidate models on accuracy, bias, and fairness to guide model selection.

Re-engineered the inference path around a persistent vLLM engine and streaming relay, replacing per-request AWS Step Functions/Batch execution — reducing time-to-first-token from 1.9 s to 23 ms (~99%), eliminating gateway timeouts, and accelerating repeat agent calls 24× through cached prompt prefixes.
Built autoscaling multi-model serving on ECS, packing multiple vLLM engines per GPU to cut per-model cost by 75%; drove scale-out/in via custom CloudWatch metrics and alarms, with health-gated production deployments.

Resolved production issues end-to-end — from root-cause analysis through fixes and regression tests, with runbooks enabling reliable handoff and ongoing maintenance.
Drove system design and code reviews across these systems, working cross-functionally with ML, product and customer teams; mentored an intern and made AI-assisted development an integral part of the engineering workflow.

Built Delta, a change-intelligence platform that maintains live state per tracked entity and emits events only for material changes.
Engineered Delta’s event-driven backbone with transactional outbox and Kafka; added Debezium CDC for analytics/reporting and deployed services on Kubernetes with Helm, Terraform, and CI.
Kept Delta’s state deterministic — constrained LLM extraction to structured output grounded in source text at ingest, while RAG chat cites retrieved evidence at query time.

LANGUAGES: Python · Java · TypeScript · JavaScript · SQL · Bash
GENAI & LLMOPS: Agno · LiteLLM · Model Context Protocol · FastMCP · Function Calling · RAG · Prompt Engineering · Context Engineering · Model Evaluation · Amazon Bedrock · Gemini
AI GOVERNANCE: Guardrails · PII Protection · Data Privacy · Hallucination Mitigation · AgentOps
ML & MLOPS: vLLM · PyTorch · CUDA · KV Cache · Model Deployment · Model Monitoring · GPU Benchmarking
RETRIEVAL & DATA: PostgreSQL · pgvector · Vector Databases · Reranking · Embeddings · Redis · MongoDB · OpenSearch
BACKEND & APIS: FastAPI · Pydantic · asyncio · SQLAlchemy · Celery · Kafka · REST · SSE · Microservices
CLOUD & DEVOPS: AWS (ECS · EC2 · Lambda · Step Functions · SQS · S3 · CloudWatch) · Kubernetes · Helm · Terraform · Docker · CI/CD · Linux