Writer
/

palmyra-mini-thinking-b-MLX-BF16

@@ -196,17 +196,27 @@ response = generate(model, tokenizer, prompt=prompt, max_tokens=200)
 Apache 2.0
 ---
-# Original model card: : palmyra-mini-thinking-b
-## Model Details
-**Model Name:** palmyra-mini-thinking-b
-**Version:** 1.0
-**Type:** Generative AI Language Model
 ## Introduction
@@ -220,44 +230,140 @@ The model's mathematical abilities are particularly noteworthy. It achieves an i
 Beyond mathematics, Palmyra-mini-thinking-b demonstrates strong performance in the competitive programming arena. Its score of 0.6343 on the Codeforces (pass_rate) benchmark underscores its ability to understand complex algorithmic problems and generate correct, efficient code. This capability suggests the model is well-suited for tasks involving code generation, debugging, and algorithmic design, making it a valuable asset for software developers and computer science researchers.
-## Benchmark Scores
-| Benchmark                                                        |    Score |
-|:-----------------------------------------------------------------|---------:|
-| gsm8k (strict-match)                                             | 0.4268   |
-| minerva_math(exact_match)                                        | 0.0708   |
-| mmlu_pro(exact_match)                                            | 0.2926   |
-| hendrycks_math                                                   | 0.0016   |
-| ifeval (inst_level_loose_acc)                                    | 0.3297   |
-| mathqa (acc)                                                     | 0.3045   |
-| humaneval (pass@1)                                               | 0.0732   |
-| BBH (get-answer)(exact_match)                                    | 0.288    |
-| mbpp                                                             | 0.168    |
-| leadboard_musr (acc_norm)                                        | 0.3796   |
-| gpqa  lighteval gpqa diamond_pass@1:8_samples                    | 0.3958   |
-| AIME24(pass@1)(avg-of-1)                                         | 0.6      |
-| AIME25(pass@1)(avg-of-1)                                         | 0.5      |
-| Livecodebench-codegen (livecodebench/code_generation_lite v4_v5) | 0.2873   |
-| AMC23                                                            | 0.925    |
-| MATH500                                                          | 0.882    |
-| Minerva                                                          | 0.2941   |
-| Olympiadbench (extractive_match)                                 | 0.5733   |
-| Codecontests (pass_rate)                                         | 0.2018   |
-| Codeforces (pass_rate)                                           | 0.6343   |
-| Taco (pass_rate)                                                 | 0.3456   |
-| APPS (all_levels)                                                | 0.0584   |
-| HMMT23 (extractive_match)                                        | 0.2333   |
-| Average                                                          | 0.359378 |
-## Intended Use
-This model is intended for research and development in the field of generative AI, particularly for tasks requiring mathematical and logical reasoning.
-## Limitations
-The model's performance has been evaluated on a specific set of benchmarks. Its performance on other tasks or in real-world applications may vary.
 ## Ethical Considerations
-As with any language model, there is a potential for generating biased or inaccurate information. Users should be aware of these limitations and use the model responsibly.

 Apache 2.0
+#### Original model card below:
 ---
+<div align="center">
+  <h1>Palmyra-mini-thinking-b</h1>
+</div>
+<p align="center">
+  <img src="https://huggingface.co/Writer/palmyra-mini-thinking-b/resolve/main/logo-mini-b%20benchmark-performance.png?download=true" width="800"/>
+</p>
+### Model Description
+- **Language(s) (NLP):** English
+- **License:** Apache-2.0
+- **Finetuned from model:** Qwen/Qwen2.5-1.5B
+- **Context window:** 131,072 tokens
+- **Parameters:** 1.7 billion
 ## Introduction
 Beyond mathematics, Palmyra-mini-thinking-b demonstrates strong performance in the competitive programming arena. Its score of 0.6343 on the Codeforces (pass_rate) benchmark underscores its ability to understand complex algorithmic problems and generate correct, efficient code. This capability suggests the model is well-suited for tasks involving code generation, debugging, and algorithmic design, making it a valuable asset for software developers and computer science researchers.
+## Benchmark Scores (sampling params: temperature:0.6, top_p:0.95)
+Pass@1(avg-of-64)
+| Benchmark | Pass@1 (avg-of-64)   | Majority@64  |
+| :-------- | :------------------- | :----------- |
+| AIME24    | 59.43%               | 71.67%       |
+| AIME25    | 49.69%               | 60.00%       |
+| GPQA      | 42.01%               | 47.22%       |
+| HMMT25    | 27.86%               | 30.00%       |
+| HLE       | 5.22%                | N/A          |
+| MMLU-PRO  | 55.49%               | 60.60%       |
+| MATH500   | 93.80%               | 95.40%       |
+| LCB       | 34.51%               | N/A          |
+LCB here is version v6_2408_2505
+Pass@1(avg-of-1)
+| Benchmark                                                        |   Score (%) |
+|:-----------------------------------------------------------------|------------:|
+| GSM8K (strict-match)                                             | 42.68%      |
+| Minerva Math (exact match)                                       | 7.08%       |
+| MMLU-PRO (exact match)                                           | 29.26%      |
+| MATH (Hendrycks)                                                 | 0.16%       |
+| IFEval (inst_level_loose_acc)                                    | 32.97%      |
+| MathQA (acc)                                                     | 30.45%      |
+| HumanEval (pass@1)                                               | 7.32%       |
+| BBH (get-answer)(exact match)                                    | 28.80%      |
+| MBPP                                                             | 16.80%      |
+| GPQA (diamond, pass@1: 8 samples)                                | 39.58%      |
+| AIME24 (pass@1)(avg-of-1)                                        | 60.00%      |
+| AIME25 (pass@1)(avg-of-1)                                        | 50.00%      |
+| Livecodebench-codegen (livecodebench/code_generation_lite v4_v5) | 28.73%      |
+| AMC23                                                            | 92.50%      |
+| MATH500                                                          | 88.20%      |
+| Minerva                                                          | 29.41%      |
+| Olympiadbench (extractive_match)                                 | 57.33%      |
+| Codecontests (pass_rate)                                         | 20.18%      |
+| Codeforces (pass_rate)                                           | 63.43%      |
+| Taco (pass_rate)                                                 | 34.56%      |
+| APPS (all_levels)                                                | 5.84%       |
+| HMMT (Feb 2025) (extractive_match)                               | 23.33%      |
+| Average                                                          | 35.94%      |
+### Use with transformers
+You can run conversational inference using the Transformers Auto classes with the `generate()` function. Here's an example:
+```py
+import torch
+from transformers import AutoTokenizer, AutoModelForCausalLM
+model_id = "Writer/palmyra-mini-thinking-b"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    torch_dtype=torch.float16,
+    device_map="auto",
+    attn_implementation="flash_attention_2",
+)
+messages = [
+      {
+        "role": "user",
+        "content": "You have a 3-liter jug and a 5-liter jug. How can you measure exactly 4 liters of water?"
+      }
+    ],
+input_ids = tokenizer.apply_chat_template(
+    messages, tokenize=True, add_generation_prompt=True, return_tensors="pt"
+)
+gen_conf = {
+    "max_new_tokens": 256,
+    "eos_token_id": tokenizer.eos_token_id,
+    "temperature": 0.3,
+    "top_p": 0.9,
+}
+with torch.inference_mode():
+    output_id = model.generate(input_ids, **gen_conf)
+output_text = tokenizer.decode(output_id[0][input_ids.shape[1] :])
+print(output_text)
+```
+## Running with vLLM
+```py
+vllm serve Writer/palmyra-mini-thinking-b
+```
+```py
+curl -X POST http://localhost:8000/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "Writer/palmyra-mini-thinking-b",
+    "messages": [
+      {
+        "role": "user",
+        "content": "You have a 3-liter jug and a 5-liter jug. How can you measure exactly 4 liters of water?"
+      }
+    ],
+    "max_tokens": 8000,
+    "temperature": 0.2
+  }'
+```
 ## Ethical Considerations
+As with any language model, there is a potential for generating biased or inaccurate information. Users should be aware of these limitations and use the model responsibly.
+### Footnotes
+- Base model: This model builds on NVIDIA's OpenReasoning-Nemotron-1.5B (`https://huggingface.co/nvidia/OpenReasoning-Nemotron-1.5B`).
+- Evaluation methodology:
+  - Pass@1 (avg-of-1): computed using `lm_eval` and `lighteval`.
+  - Pass@1 (avg-of-64) and Majority@64: computed using `nemoskills`.
+### Citation and Related Information
+To cite this model:
+```
+@misc{Palmyra-mini-thinking-b,
+  author = {Writer Engineering team},
+  title = {{Palmyra-mini: A powerful LLM designed for math and coding}},
+  howpublished = {\url{https://dev.writer.com}},
+  year = 2025,
+  month = Sep
+}
+```
+Contact Hello@writer.com