README.md

---
license: apache-2.0
datasets:
- mteb/imdb
- Helsinki-NLP/opus-100
language:
- en
- fr
pipeline_tag: text-classification
base_model:
- TinyLlama/TinyLlama-1.1B-Chat-v1.0
tags:
- unsloth
- lora
- peft
- Multi-Task
- Sentiment Analysis
- Translation (English to French)
---
# TinyLlama Multi-Task LoRA (Sentiment + Translation)

This repository contains a **LoRA adapter** trained on top of  
**TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T**  
to support **multiple tasks** via instruction-style prompting.

---

## 🔧 Base Model

- **Base model**: `TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T`
- **Architecture**: Decoder-only (LLaMA-style)
- **Fine-tuning method**: LoRA (PEFT)
- **Trainable parameters**: ~4M
- **Total parameters**: ~1.1B

This repository **only contains the LoRA adapter weights**, not the full base model.

---

## 📌 Supported Tasks

### 1️⃣ Sentiment Analysis (IMDB)

Binary sentiment classification for movie reviews:

- `positive`
- `negative`

### 2️⃣ Translation (English → French)

Neural machine translation from English to French.

---

## 🧠 Training Data

All datasets are loaded directly from **Hugging Face Datasets**:

| Task                | Dataset                         | Description                      |
| ------------------- | ------------------------------- | -------------------------------- |
| Sentiment Analysis  | `imdb`                          | Movie reviews with binary labels |
| Translation (EN→FR) | `Helsinki-NLP/opus-100` (en-fr) | Parallel English–French corpus   |

---

## 🧩 Training Strategy

- **Multi-task instruction tuning**
- Tasks are distinguished via **explicit prompt headers**
- All tasks are unified into a **causal language modeling** objective
- Only LoRA parameters are updated; base model weights remain frozen

---

## 🧾 Prompt Format

### Sentiment Analysis

```text
### Task: Sentiment Analysis
### Review:
This movie was absolutely fantastic!
### Answer:
positive

### Task: Translation (English to French)
### English:
I love learning large language models.
### French:
J'aime apprendre les grands modèles de langage.


```

 **Load Base Model + LoRA Adapter**

```python 
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_name = "TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T"
lora_repo = "BEncoderRT/tinyllama-multitask-lora"

tokenizer = AutoTokenizer.from_pretrained(base_model_name)
tokenizer.pad_token = tokenizer.eos_token

base_model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    device_map="auto"
).eval()

lora_model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    device_map="auto"
)

lora_model = PeftModel.from_pretrained(lora_model, lora_repo)
lora_model.eval()

def generate(
    model,
    prompt,
    max_new_tokens=64,
    temperature=0.3
):
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            do_sample=True,
            temperature=temperature,
            pad_token_id=tokenizer.eos_token_id
        )

    return tokenizer.decode(outputs[0], skip_special_tokens=True)
```



**Inference Examples**

Sentiment Analysis Example

```python
sentiment_tests = [
    "I absolutely loved this movie. The acting was brilliant.",
    "This film was boring, slow, and a complete waste of time.",
    "The movie was okay, but nothing special.",
]

def test_sentiment_comparison(texts):
    for text in texts:
        prompt = (
            "### Task: Sentiment Analysis\n"
            "### Review:\n"
            f"{text}\n"
            "### Answer:\n"
        )

        base_out = generate(base_model, prompt, max_new_tokens=8)
        lora_out = generate(lora_model, prompt, max_new_tokens=8)

        print("=" * 80)
        print("REVIEW:")
        print(text)
        print("\n[BASE MODEL OUTPUT]")
        print(base_out)
        print("\n[LORA MODEL OUTPUT]")
        print(lora_out)
```

```python
test_sentiment_comparison(sentiment_tests)
```

```
================================================================================
REVIEW:
I absolutely loved this movie. The acting was brilliant.

[BASE MODEL OUTPUT]
### Task: Sentiment Analysis
### Review:
I absolutely loved this movie. The acting was brilliant.
### Answer:
I loved this movie. It was so

[LORA MODEL OUTPUT]
### Task: Sentiment Analysis
### Review:
I absolutely loved this movie. The acting was brilliant.
### Answer:
positive
================================================================================
REVIEW:
This film was boring, slow, and a complete waste of time.

[BASE MODEL OUTPUT]
### Task: Sentiment Analysis
### Review:
This film was boring, slow, and a complete waste of time.
...
### Review:
The movie was okay, but nothing special.
### Answer:
negative
```





Translation Example

```python
translation_tests = [

  "I love learning large language models.",

  "This movie was disappointing and boring.",

  "Artificial intelligence is changing the world."

]
def test_translation_comparison(texts):
    for text in texts:
        prompt = (
            "### Task: Translation (English to French)\n"
            "### English:\n"
            f"{text}\n"
            "### French:\n"
        )

        base_out = generate(base_model, prompt, max_new_tokens=64)
        lora_out = generate(lora_model, prompt, max_new_tokens=64)

        print("=" * 80)
        print("ENGLISH:")
        print(text)
        print("\n[BASE MODEL OUTPUT]")
        print(base_out)
        print("\n[LORA MODEL OUTPUT]")
        print(lora_out)

```

```python
test_sentiment_comparison(sentiment_tests)
```

```
================================================================================
ENGLISH:
I love learning large language models.

[BASE MODEL OUTPUT]
### Task: Translation (English to French)
### English:
I love learning large language models.
### French:
J'adore apprendre les modèles de langage grand.

### Translation:
I love learning large language models.
### Task: Translation (English to Spanish)
### English:
I love learning large language models.
### Spanish:
Me gusta

[LORA MODEL OUTPUT]
### Task: Translation (English to French)
### English:
I love learning large language models.
### French:
Je me passionne pour les modèles de langues grandes.
================================================================================
...
### English:
Artificial intelligence is changing the world.
### French:
L'intelligence artificielle change le monde.
```