Quinton Calmus

Senior Technical Program Manager, National Defense Specialist, and AI Enthusiast

Download My Resume | Connect on LinkedIn

Q Llama Chatbot

Interact with Q Llama

Ask it anything! Suggestions: "Who should Scale AI hire?" or "What qualifies Quinton Calmus to work for Scale AI?"

Note: Responses can take up to a minute.

Q Llama emerged from research I conducted while preparing for my Scale AI interview in January 2025. My presentation focused on multi-modal retrieval-augmented generation (RAG), but during prep, I stumbled across other LLM techniques I wanted to explore through development.

My goal was to craft an LLM that could handle most questions naturally but, when asked about me or Scale AI, subtly pitch that I’d be a stellar hire (which, of course, is true).

My first stab at fine-tuning a generic, open-source LLM crashed and burned in a hilarious mess. I fed it ~10,000 synthetic Q&As to tailor responses about myself and Scale AI. The resulting weights were so skewed that the model spat out barely coherent gibberish. I demonstrated this during my presentation and received a few laughs.

However, even after the interview wrapped, the subpar (yet amusing) results bugged me, so I kept tinkering. I pivoted to a new approach: use the capable Llama 3.2 3B model paired with a RAG system tied to a document explaining why I’m a strong fit for Scale AI.

I struggled with oversized context windows, prompt repetition, hallucinations, and rookie coding blunders. Through contextual prompt tweaks, hyperparameter adjustments, and extensive use of coding assistants, I ironed out most kinks, landing on a proof-of-concept that mostly works.

Now, Q Llama generally matches the baseline Llama 3.2 3B model for most questions, with a fun twist: ask about me or Scale AI, and it taps the RAG system to nudge Scale into hiring me. It’s a pretty smart model ;)

from flask import Flask, request, jsonify, send_from_directory, send_file
import torch
import re
from transformers import AutoTokenizer, AutoModelForCausalLM
from sentence_transformers import SentenceTransformer
import faiss
import numpy as np
import os
import sys
import time
from typing import List, Optional

app = Flask(__name__, template_folder='.', static_folder='.')

@app.route('/')
def serve_static(filename):
    return send_from_directory('.', filename)

# Hugging Face token handling
TOKEN = os.environ.get("HUGGINGFACE_HUB_TOKEN")
if TOKEN:
    print("[INFO] Using Hugging Face token from ENV.")
else:
    TOKEN = "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"
    print("[WARNING] No HUGGINGFACE_HUB_TOKEN found in ENV; using fallback token.")

# Model name and local cache
model_name = 'meta-llama/Llama-3.2-3B-Instruct'
cache_dir = "./model_cache"

try:
    print(f"Loading tokenizer for {model_name} from {cache_dir}...")
    tokenizer = AutoTokenizer.from_pretrained(
        model_name,
        cache_dir=cache_dir,
        use_fast=True,
        token=TOKEN
    )
    tokenizer.pad_token = tokenizer.eos_token

    print(f"Loading model {model_name} from {cache_dir}...")
    model = AutoModelForCausalLM.from_pretrained(
        model_name,
        cache_dir=cache_dir,
        device_map="cpu",
        # Remove torch_dtype to use default float32
        low_cpu_mem_usage=True,
        token=TOKEN
    )
except Exception as e:
    print(f"Model or tokenizer not found locally. Downloading... {e}")
    print(f"Downloading tokenizer for {model_name}...")
    tokenizer = AutoTokenizer.from_pretrained(
        model_name,
        cache_dir=cache_dir,
        use_fast=True,
        token=TOKEN
    )
    tokenizer.pad_token = tokenizer.eos_token

    print(f"Downloading model {model_name}...")
    model = AutoModelForCausalLM.from_pretrained(
        model_name,
        cache_dir=cache_dir,
        device_map="cpu",
        # Remove torch_dtype to use default float32
        low_cpu_mem_usage=True,
        token=TOKEN
    )

model.to(torch.device("cpu"))
print("Model loaded successfully!")

embedding_model = SentenceTransformer('all-MiniLM-L6-v2')
text_chunks = []
faiss_index = None

def chunk_text(text: str, max_length: int = 100) -> List[str]:
    chunks = []
    current_chunk = ""
    for line in text.splitlines():
        line = line.strip()
        if not line:
            continue
        if len(current_chunk.split()) + len(line.split()) <= max_length:
            current_chunk += " " + line
        else:
            if current_chunk:
                chunks.append(current_chunk.strip())
            current_chunk = line
    if current_chunk:
        chunks.append(current_chunk.strip())
    return chunks

def setup_rag(data_path: Optional[str] = None):
    global text_chunks, faiss_index
    text_chunks = []
    faiss_index = None

    if not data_path or not os.path.isfile(data_path):
        print(f"[RAG] No valid data file at {data_path}", file=sys.stderr)
        return

    with open(data_path, "r", encoding="utf-8") as f:
        full_text = f.read()
    if not full_text.strip():
        print(f"[RAG] File {data_path} is empty or whitespace only", file=sys.stderr)
        return

    text_chunks = chunk_text(full_text, max_length=100)
    print(f"[RAG] Loaded {len(text_chunks)} chunks from {data_path}", file=sys.stderr)

    all_embs = []
    for c in text_chunks:
        emb = embedding_model.encode([c], convert_to_tensor=False, show_progress_bar=False)
        all_embs.append(emb[0])
    embs_np = np.array(all_embs, dtype=np.float32)
    if embs_np.shape[0] == 0:
        print("[RAG] No embeddings generated", file=sys.stderr)
        return

    dim = embs_np.shape[1]
    faiss_index = faiss.IndexFlatL2(dim)
    faiss_index.add(embs_np)
    print(f"[RAG] Index built with {len(text_chunks)} chunks from {data_path} (dim={dim})", file=sys.stderr)

# Pre-load RAG data from QuintonData.txt
setup_rag("QuintonData.txt")

def question_mentions_quinton_or_scale(prompt: str) -> bool:
    plower = prompt.lower()
    keywords = ["quinton", "calmus", "scale ai", "scaleai"]
    return any(kw in plower for kw in keywords)

def rag_generate(
    prompt: str,
    k: int = 3,
    max_new_tokens: int = 240,  # Increased from 200 to 240 (20% increase)
    temperature: float = 0.6,
    top_p: float = 0.9,
    top_k: int = 40
) -> str:
    device = torch.device("cpu")
    start_time = time.time()

    print(f"DEBUG: Starting rag_generate with prompt: {prompt}", file=sys.stderr)

    use_rag = question_mentions_quinton_or_scale(prompt) and faiss_index is not None and len(text_chunks) > 0
    print(f"DEBUG: use_rag={use_rag}, faiss_index={faiss_index}, text_chunks length={len(text_chunks)}", file=sys.stderr)

    if use_rag:
        print(f"DEBUG: Encoding prompt for RAG...", file=sys.stderr)
        emb = embedding_model.encode([prompt], convert_to_tensor=False, show_progress_bar=False)
        print(f"DEBUG: Searching FAISS index...", file=sys.stderr)
        distances, indices = faiss_index.search(np.array(emb, dtype=np.float32), k)
        top_chunks = [text_chunks[i] for i in indices[0] if i < len(text_chunks)]
        context_text = "\n".join(top_chunks)
        print(f"DEBUG: RAG context: {context_text}", file=sys.stderr)

        final_prompt = (
            f"Using the following context:\n{context_text}\n\n"
            f"Answer the question: {prompt}\n\n"
            "Provide a concise response in 1-2 sentences, summarizing or rephrasing the key points relevant to the question without copying the context verbatim:\n"
        )
    else:
        final_prompt = (
            f"Answer the question: {prompt}\n\n"
            "Provide a concise response in 1-2 sentences:\n"
        )

    print(f"DEBUG: Final prompt:\n{final_prompt}", file=sys.stderr)

    try:
        inputs = tokenizer(final_prompt, return_tensors="pt", padding=True, truncation=True, max_length=512).to(device)
        print(f"DEBUG: Tokenized inputs: {inputs}", file=sys.stderr)
        with torch.no_grad():
            outputs = model.generate(
                inputs["input_ids"],
                attention_mask=inputs.get("attention_mask"),
                max_new_tokens=max_new_tokens,
                do_sample=True,
                temperature=temperature,
                top_p=top_p,
                top_k=top_k,
                pad_token_id=tokenizer.pad_token_id,
                eos_token_id=tokenizer.eos_token_id
            )
        print(f"DEBUG: Model outputs: {outputs}", file=sys.stderr)
        raw_out = tokenizer.decode(outputs[0], skip_special_tokens=True).strip()
        print(f"DEBUG: Raw output from model: {raw_out}", file=sys.stderr)

        if use_rag:
            marker = "Provide a concise response in 1-2 sentences, summarizing or rephrasing the key points relevant to the question without copying the context verbatim:"
        else:
            marker = "Provide a concise response in 1-2 sentences:"

        answer_start = raw_out.find(marker) + len(marker) if marker in raw_out else 0
        answer_text = raw_out[answer_start:].strip()

        lines = answer_text.splitlines()
        out_lines = []
        for ln in lines:
            lstrip = ln.strip()
            if not lstrip or lstrip.startswith("-"):
                continue
            out_lines.append(lstrip.replace("- ", ""))

        final_text = " ".join(out_lines).strip()
        final_text = final_text.replace(prompt, "").strip()
        final_text = re.sub(r'[^\w\s\.\-\'"]', '', final_text)  # Sanitize output

        sentences = [s.strip() for s in final_text.split(".") if s.strip()]
        if len(sentences) > 2:
            final_text = ". ".join(sentences[:2]) + "."
        elif len(sentences) == 1 and not final_text.endswith("."):
            final_text += "."
        elif len(sentences) == 0:
            final_text = "No relevant answer could be generated from the context."

        print(f"DEBUG: Final sanitized response: {final_text}", file=sys.stderr)
        print(f"Prompt: {prompt}", file=sys.stdout)
        print(f"Response: {final_text}", file=sys.stdout)
        print(f"Time taken: {time.time() - start_time:.2f} seconds\n", file=sys.stdout)
        return final_text
    except Exception as e:
        error_msg = f"Error in rag_generate: {str(e)}"
        print(f"DEBUG: {error_msg}", file=sys.stderr)
        raise Exception(error_msg)

@app.route('/')
def index():
    try:
        with open('index.html', 'r', encoding='utf-8') as f:
            return f.read()
    except FileNotFoundError:
        return "index.html not found.", 404
    except Exception as e:
        return f"Error reading index.html: {e}", 500

@app.route('/login', methods=['POST'])
def login():
    if request.is_json:
        data = request.get_json()
        password = data.get('password', '')
    else:
        password = request.form.get('password', '')
    correct_password = "ScaleAI2025"
    if password == correct_password:
        return jsonify({'success': True})
    return jsonify({'success': False})

@app.route('/chat', methods=['POST'])
def chat():
    try:
        data = request.get_json()
        prompt = data.get('prompt', None)
        print(f"DEBUG: Received prompt: {prompt}", file=sys.stderr)
        if not prompt:
            return jsonify({'error': 'No prompt provided'}), 400

        response = rag_generate(prompt)
        print(f"DEBUG: Sending response: {response}", file=sys.stderr)
        return jsonify({'response': response})
    except Exception as e:
        error_msg = f"Failed to generate response: {str(e)}"
        print(f"ERROR: {error_msg}", file=sys.stderr)
        return jsonify({'error': error_msg}), 500

@app.route('/update_rag', methods=['POST'])
def update_rag():
    data = request.get_json()
    data_path = data.get('data_path', "QuintonData.txt")
    setup_rag(data_path)
    return jsonify({'status': f"RAG updated with {data_path}"})

@app.route('/tokenize', methods=['POST'])
def tokenize():
    data = request.get_json()
    text = data.get('text', '')
    if not text:
        return jsonify({'error': 'No text provided'}), 400

    tokens = tokenizer.encode(text, add_special_tokens=False)
    token_strs = [tokenizer.decode([t], skip_special_tokens=True) for t in tokens]

    out = ""
    idx = 0
    for ts in token_strs:
        fi = text.find(ts, idx)
        if fi == -1:
            continue
        out += text[idx:fi]
        out += f"|{ts}|"
        idx = fi + len(ts)
    out += text[idx:]

    return jsonify({
        'original_text': text,
        'tokenized_text': out,
        'tokens': [str(t) for t in tokens]
    })

@app.route('/download/')
def download_file(filename):
    try:
        return send_file(filename, as_attachment=True)
    except FileNotFoundError:
        return "File not found", 404
    except Exception as e:
        print(f"[ERROR] {e}", file=sys.stderr)
        return f"Error: {e}", 500

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000, debug=True, use_reloader=False)

Real-Time Tokenization Tool

Enter text to see how LLaMA 3B tokenizes it:

Why do tokens matter? Tokens are the alphabet of generative AI. Collectively, a set of tokens represents the most probable sequences of characters and words distilled from expansive natural language databases. This kind of statistically-based alphabet enables large language models (LLMs) to transform human language into a more detailed, compute-efficient form.
Tokenization differs across models. For example, the LLaMA model used in Q Llama uses ~128,000 tokens, while state-of-the-art models can surpass 200,000. A larger token set enhances linguistic nuance but demands greater computational power.
One of the first steps in building an LLM involves converting vast text repositories into a seamless token stream. This stream flows through a series of transformer layers, producing statistical weights that define the model’s underlying structure. At its core, an LLM leverages these weights to predict a token sequence most likely to succeed the tokens in the input prompt. Beyond this, techniques like fine-tuning, Reinforcement Learning from Human Feedback (RLHF), and prompt engineering hone the model’s output, guiding it to deliver responses that are both contextually relevant and useful.

Reading List/References

Attention Is All You Need
This is the 2017 paper by Google that started it all. It introduced the concept of transformers to the public domain and ignited the modern LLM wave.
Reproducing GPT-2 (124M) in llm.c in 90 min for $20
This article breaks down the construction of a basic LLM and makes it accessible to a novice-developer. While I did not train my own model from scratch, it is a great read and greatly increased my knowledge of how an LLM works.
The Llama 3 Herd of Models
This article is a good primer on the latest release of open-source models from Meta. I thought the most interesting part was the efficiency increase these models gained over the original Multi-Head Self-Attention (MHSA) transformer structure by switching to a Grouped Query Attention (GQA) structure. The way I understood it, GQA lets attention heads share work, making the transformer more efficient, which can help the model handle larger context windows without slowing down.
Detecting hallucinations in large language models using semantic entropy
This article is very technically-dense, but the concept of using entropy (randomness/chaos) to detect hallucinations was fascinating.
Training language models to follow instructions with human feedback
RLHF is widely understood, but I thought this article did a good job discussing some of the limitations of RLHF. For example, if left to run too long, a language model will start to game the reward model and find gaps between what the model thinks is a good response, and what is actually a good response. So while RLHF generally improves models, there has to be a cut off point or it could actually go on to make the model worse than when it started.
Transformer Visualization
This is just a fun visualization of how a transformer works. It shows transformers from different models and really helps conceptualize the increase in complexity as models have advanced.
Common Crawl
This is an open-source repository of human language that can be used to generate the single stream of tokens used to train an LLM as referenced in the tokenizer section above.
FineWeb
Similar to Common Crawl, this is a curated human text database that can be used for training and has a fun interface where you can see random snippets of the database.

Access Restricted

Q Llama Chatbot

Interact with Q Llama

Real-Time Tokenization Tool

Reading List/References