Project Goal

Build a public, portfolio-grade AI assistant that demonstrates real-world engineering around LLM applications: LLM pipelines, vector retrieval, streaming SSE, cloud deployment, rate limiting, cost-aware guardrails, API design, and CI/CD + infrastructure-as-code.

High-Level Architecture

Request/response flow:

• Frontend (GitHub Pages) → chat UI sends a user message
• API Gateway → routes requests
• AWS Lambda → runs the application
• FastAPI app → orchestrates retrieval + generation
• Retrieval service → top-k chunk selection + reranking
• Vector index in S3 → downloaded on cold start and loaded into memory
• LLM → generates a grounded answer
• SSE stream → tokens are streamed back progressively

Backend API Design

• POST API for query: primary endpoint returning Server Sent Events (SSE) for streaming responses.
• GET API for health: health checks for deployment/monitoring.
• GET API to read metrics: lightweight service metrics.
• POST API to Reindex documents: rebuilds the vector index from source documents.

RAG Pipeline (How Answers Are Grounded)

The system follows the standard RAG pattern: retrieve relevant knowledge first, then generate an answer using that context. This is specifically intended to reduce hallucinations compared to an LLM-only approach.

• Ingestion: CV, project descriptions, Kaggle achievements, publications, GitHub content, and profile text are cleaned and chunked.
• Embedding: each chunk becomes a vector embedding.
• Index: the vector index is persisted to cloud storage and loaded into memory on cold start.
• Retrieval: query → embedding → vector similarity search → top-k chunks → context assembly.
• Generation: system prompt + retrieved chunks + user query → LLM answer.

Streaming Responses (SSE)

Responses are streamed back to the browser as SSE, so the user sees tokens as they are generated. This improves perceived latency and makes the assistant feel responsive. Typical SSE events include incremental chunks and a completion event (e.g., event: chunk, event: done).

Cost & Abuse Controls

• Rate limiting: request caps (e.g., per-minute/per-hour) prevent abuse and keep costs predictable.
• Spend guardrails: daily/monthly budget controls (e.g., DynamoDB-backed in production, in-memory locally) protect against runaway costs.
• Rate limit headers: the API can return standard limit/remaining/reset headers for transparency.

Observability

The system emits operational signals (e.g., rate-limit decisions and streaming handler errors) to Amazon CloudWatch to support debugging and production monitoring.

Deployment & CI/CD

• Infrastructure-as-Code: AWS resources are provisioned via AWS CDK (Lambda, API Gateway, S3, permissions).
• GitHub Actions: automated lint/security/tests + CDK synth + deploy pipelines.
• Cold start strategy: vector index stored in S3 and loaded into memory on startup avoids recomputing embeddings on each deploy.

System Metrics

• Retrieval latency: typically ~100-300ms depending on index size and cold starts.
• Generation latency: depends on the LLM and response length, typically ~700-950ms.
• Streaming chunk size: tokens are sent in small batches (e.g., 20-50 tokens) for responsiveness.
• Rate limits: e.g., 5 requests/minute, 20 requests/hour (configurable).
• Cost per request: varies based on LLM usage and retrieval, but guardrails keep it within a predictable range.

Why This Project Is Valuable

This is an AI systems engineering project, not just a model demo. It demonstrates the practical building blocks needed for modern LLM products: grounded retrieval, streaming UX, cost controls, cloud-native deployment, and operational readiness.