A path to be an ML Engineer

MailSense is a project that I built to learn ML Engineering, it predicts whether an email will be ignored, read, or acknowledged, a 3-class classification problem on real email campaign data (~68,000 emails). Dataset: Email Campaign Management for SME.

Then it:

1. Outputs a calibrated engagement score 0-100 (score 70 means ~70% chance of engagement)
2. Explains why with SHAP feature contributions
3. Suggests improvements via a fine-tuned LLM grounded in the same SHAP factors

Before you read

Terms I will use:

• Model: a program that learns patterns from examples. Show the program 68,000 emails with their outcomes, it figures out what makes an email get read or ignored.
• Training: the process of showing the model examples so it learns. Like teaching a kid to recognize animals by showing pictures.
• Feature: one piece of information the model uses. Word count is a feature. Number of links is a feature. The subject line hotness score is a feature.
• LLM (Large Language Model): a model trained on text that can read and write. ChatGPT is an LLM. Qwen is a smaller, free LLM I used in this project.
• JSON: a structured text format that looks like this: {"name": "Agam", "score": 70}. Computers read it easily. Humans can too.

Now the actual learnings.

• One-Hot Encoding: turning a category into a yes/no column, separate column for each category, like taking one color column and splitting it into is_red / is_blue / is_green columns.
• Target Encoding: replacing a category with a number based on the outcome you are trying to predict. Instead of just Location A or Location B, you use the average engagement rate for each location.
• Classifier is a model that puts data into categories. In this project, it reads an email's features and decides which of the three it belongs to: ignored, read, or acknowledged.

What I learned

1. XGBoost
XGBoost is a gradient boosting library, it builds decision trees one after another, each new tree focusing on the mistakes the previous trees made.

Think of it like an apprentice learning from a master craftsman, iteration by iteration. The first attempt is rough. The master points out the errors. The apprentice tries again, this time paying extra attention to what went wrong. After many rounds, the work is precise, not because any single attempt was perfect, but because each round fixed a little bit more.

A single decision tree is like a flowchart: if word_count > 500, then check total_links, otherwise check subject_hotness. One tree alone is too simple, it either memorizes the training examples without understanding patterns, or it is too vague to be useful. But 200 trees, each correcting the previous one's errors, make a strong team.

The dataset had 68,000 rows of tabular data with 32 features: numeric columns like Word_Count and Total_Links, yes/no flags like is this email type 1?, location-based numbers like what is the average engagement for this customer's region?, and combination columns like past communications x campaign type.

2. Optuna
Optuna is a framework for tuning those knobs automatically. Every model has settings you can adjust. How fast it learns. How deep each tree grows. How much it penalizes complexity. Finding the right combination by hand is just trial and error.

Think of it like cooking a dish with 6 ingredients, each with a different amount. You could try every possible combination one by one (GridSearch). That works if you have 2 ingredients. With 6 ingredients and 5 possible amounts each, you have 15,625 combinations, and each one takes minutes to taste (train). That is GridSearch, thorough but impractical.

RandomSearch picks combinations at random. Faster, but you might skip the good ones by chance.

Optuna uses an approach called Bayesian search. It remembers which settings worked and which did not, then tries new combinations similar to the good ones and avoids regions that performed poorly. It is like a chef who learns from each attempt, more salt worked last time, less sugar next. With 30 trials, it found an XGBoost configuration that made the model noticeably better at sorting the rare emails correctly.

3. SHAP
SHAP is a method that answers: why did the model give this email a score of 23? It assigns a number to each feature showing how much that feature helped or hurt the prediction.

Think of it like an itemized restaurant bill. The total (prediction) is the final price, and each line item (feature) shows exactly how much that ingredient added or subtracted. A steak adds +$25. A dessert adds +$8. A discount coupon subtracts -$10. SHAP does the same for predictions: Subject_Hotness_Score contributed +0.15 to the engagement score, Word_Count subtracted -0.22.

There are two ways to look at SHAP. You can ask across all 68,000 emails, which features matter most?, like knowing that protein is always the biggest cost driver in any restaurant. Or you can ask why did this one specific email get a score of 23?, like understanding why your particular bill came out to $47. The first is the big-picture view, the second zooms into one prediction.

In this project, the /explain endpoint takes an email and returns the top 10 features that pushed its score up or down. A user can see exactly why their draft might get ignored, not just a score, but the reasons behind it.

4. Calibration
Calibration is the process of making sure a model's predicted probabilities match actual reality. A classifier can output a number like 0.60 and call it probability of engagement. But that number might not mean what it claims. Calibration fixes that.

Think of a weather app. The app says 60% chance of rain every day for 100 days. If it actually rains on 60 of those 100 days, the app is well calibrated. Its 60% genuinely means 60%. If it rains on only 30 days, the app is overconfident. Its 60% really means 30%. Calibration is what turns the second app into the first.

Before calibration, my XGBoost was the overconfident app. When it predicted 94% chance of engagement, only 69% of those emails actually got engaged. The model was confident but wrong.

The metric that measures this is Expected Calibration Error (ECE). You take all predictions, group them into 10 buckets (0-10%, 10-20%, ..., 90-100%), and for each bucket compare the average predicted probability to the actual observed frequency. The weighted average of these gaps is the ECE. My uncalibrated model had an ECE of 0.226, on average, predictions were off by 22 percentage points.

The fix is isotonic regression. It looks at the model's predictions and learns a correction curve, like adjusting a scale that always reads 10% too high. You feed it examples of the model said 60%, but the real answer was 42%, and it learns to adjust every future prediction accordingly. After calibration, ECE dropped to 0.0086, a 96% reduction. Predictions are now off by less than 1 percentage point on average.

The score 0-100 returned by the API is the calibrated probability x 100. Score 70 genuinely means ~70% chance of engagement. Not just a number the model spat out, a number that has been verified against reality.

5. Data leakage
Sometimes a model can cheat without you realizing it. In my dataset, I had a column called Customer_Location, the customer's region (A through G). Instead of using A or B directly, I replaced it with a number: the average engagement rate for each region. Region A averages 24% engagement, Region G averages 23%. This gives the model more useful information than just the letter.

The danger: if I compute these averages using ALL the data including the test set, the model sees answers it should not have access to yet. This is like studying for an exam with the answer key open, you score perfectly, but only because you cheated.

The fix is simple. Split the data first. Compute the averages only from the training portion. Then apply those same numbers to the test portion. The test data is never used to calculate anything, it only receives numbers computed from the training data alone. No information leaks across the boundary.

6. QLoRA and LoRA
LoRA is a technique for updating a large model without changing all of it. Instead of rewriting the model's entire 500-million-parameter brain, you add a tiny 10MB attachment, like sticking a Post-it note next to each page of a thick encyclopedia rather than rewriting the book.

I uploaded the LoRA adapter to HuggingFace: dandriaalabasta/mailsense-assistant-lora.

QLoRA adds compression on top. It shrinks the model's precision from 16 decimal places to roughly 4, cutting memory use from 2GB to about 500MB. A free Google Colab GPU can handle this easily.

After training, I merged the adapter into the base model, a 1.9GB standalone file that runs on a normal CPU. I uploaded the merged model to HuggingFace: dandriaalabasta/mailsense-assistant-merged.

The alternative is full fine-tuning (rewriting the entire encyclopedia, expensive) or RAG (keeping a reference book open during the exam, no permanent learning). QLoRA was the right fit: training took 30 minutes on a free GPU, and the merged model runs standalone without needing the adapter or any special loading.

7. Fine-tuning vs RAG
RAG works like an open-book exam: the model can look up information when it needs it, but it never actually learns anything permanently. The answers come from searching external documents, not from the model's own understanding.

Fine-tuning is like memorizing the multiplication table. The model's internal wiring changes so the knowledge becomes part of it. No need to look anything up, it just knows.

For MailSense, I chose fine-tuning. The task is narrow (generate email suggestions), the training data is structured (640 example conversations), and the fine-tuned model runs on its own without depending on any external search system or paid API.

8. Tool calling and agentic behavior
Tool calling means the LLM can use other programs to get information it does not have. A chef who does not know the temperature outside steps out to check before deciding whether to serve hot soup or cold gazpacho. The chef did not memorize the weather, they used a tool (stepping outside).

Agentic means the LLM follows a plan on its own: look at the email, figure out what is wrong, draft fixes, then check if the fixes actually work. Nobody tells it each step, it decides the sequence.

In MailSense, the /assistant endpoint does exactly this. It first calls /explain to understand which features are hurting the score. Then it generates suggestions that specifically reference those features. Then it calls /score on the improved version to predict how much the score would go up. The suggestions are grounded in real model analysis, not invented out of thin air.

9. Evaluation harness with LLM-as-judge
I used DeepSeek as an impartial judge to compare the original Qwen model against my fine-tuned version. Think of it like a blind taste test, the judge does not know which response came from which model, it just scores each one on the same criteria.

The judge evaluated 20 pairs of responses, prompts the models had never seen during training. Four criteria: are the suggestions grounded in SHAP factors? Are they concrete enough to act on? Does the reasoning make sense? Is the response properly formatted as JSON?

The fine-tuned model won 56% of head-to-head comparisons. But the strongest result: 100% of fine-tuned outputs were valid JSON, versus only 10% for the base model. The original Qwen responds in natural paragraphs. The fine-tuned version learned to structure its answers properly.

Fine-tuning proof

Metric	Original Qwen	After fine-tuning
Valid JSON output	2/20 (10%)	20/20 (100%)
Grounded to SHAP	8/20 (40%)	10/20 (50%)
LLM-as-judge win rate	35%	56%

Full report: report.md

API

Method	Endpoint	What it does
POST	/score	Predict engagement. Returns score 0-100, label, probabilities, derived stats.
POST	/explain	SHAP explanation, same request as /score. Top-10 feature contributions with +/- direction.
POST	/assistant	Improvement suggestions grounded in SHAP factors. Runs local Qwen + DeepSeek fallback.
GET	/health	{"status":"ok","model_loaded":true}
GET	/model-info	Model version, feature list, metrics (macro-F1, PR-AUC, ECE).
GET	/evaluate	Live prediction stats from SQLite log.

Repository: github.com/agmadt/ml-engineer

Live API: nalar-mail.adhityoagam.com