Upload model checkpoint

README.md

@@ -0,0 +1,199 @@
+---
+library_name: transformers
+tags: []
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]

config.json

@@ -0,0 +1,56 @@
+{
+  "_name_or_path": "/media/cheliu/xiaohang/mdpo/checkpoints/LLaDA-8B-Instruct-MDPO-numina-adv-256st-8sample_temp0.4_8gpus",
+  "activation_type": "silu",
+  "alibi": false,
+  "alibi_bias_max": 8.0,
+  "architectures": [
+    "LLaDAModelLM"
+  ],
+  "attention_dropout": 0.0,
+  "attention_layer_norm": false,
+  "attention_layer_norm_with_affine": true,
+  "auto_map": {
+    "AutoConfig": "configuration_llada.LLaDAConfig",
+    "AutoModel": "GSAI-ML/LLaDA-8B-Instruct--modeling_llada.LLaDAModelLM",
+    "AutoModelForCausalLM": "GSAI-ML/LLaDA-8B-Instruct--modeling_llada.LLaDAModelLM"
+  },
+  "bias_for_layer_norm": false,
+  "block_group_size": 1,
+  "block_type": "llama",
+  "d_model": 4096,
+  "embedding_dropout": 0.0,
+  "embedding_size": 126464,
+  "eos_token_id": 126081,
+  "flash_attention": false,
+  "include_bias": false,
+  "include_qkv_bias": false,
+  "init_cutoff_factor": null,
+  "init_device": "meta",
+  "init_fn": "mitchell",
+  "init_std": 0.02,
+  "input_emb_norm": false,
+  "layer_norm_type": "rms",
+  "layer_norm_with_affine": true,
+  "mask_token_id": 126336,
+  "max_sequence_length": 4096,
+  "mlp_hidden_size": 12288,
+  "mlp_ratio": 4,
+  "model_type": "llada",
+  "multi_query_attention": null,
+  "n_heads": 32,
+  "n_kv_heads": 32,
+  "n_layers": 32,
+  "pad_token_id": 126081,
+  "precision": "amp_bf16",
+  "residual_dropout": 0.0,
+  "rms_norm_eps": 1e-05,
+  "rope": true,
+  "rope_full_precision": true,
+  "rope_theta": 500000.0,
+  "scale_logits": false,
+  "torch_dtype": "float32",
+  "transformers_version": "4.49.0",
+  "use_cache": true,
+  "vocab_size": 126464,
+  "weight_tying": false
+}

configuration_llada.py

@@ -0,0 +1,463 @@
+"""
+LLaDA configuration
+"""
+from transformers import AutoConfig, PretrainedConfig
+
+from enum import Enum
+from os import PathLike
+from typing import Union
+from dataclasses import asdict, dataclass, field
+from glob import glob
+from pathlib import Path
+from typing import (
+    Any,
+    Dict,
+    Iterable,
+    List,
+    Optional,
+    Tuple,
+    Type,
+    TypeVar,
+    Union,
+    cast,
+)
+
+
+__all__ = [
+    "ActivationType",
+    "ActivationCheckpointingStrategy",
+    "BlockType",
+    "LayerNormType",
+    "InitFnType",
+    "ModelConfig",
+]
+
+PathOrStr = Union[str, PathLike]
+
+
+class StrEnum(str, Enum):
+    """
+    This is equivalent to Python's :class:`enum.StrEnum` since version 3.11.
+    We include this here for compatibility with older version of Python.
+    """
+
+    def __str__(self) -> str:
+        return self.value
+
+    def __repr__(self) -> str:
+        return f"'{str(self)}'"
+
+
+class LayerNormType(StrEnum):
+    default = "default"
+    """
+    The default LayerNorm implementation, equivalent to PyTorch's built-in version.
+    """
+
+    low_precision = "low_precision"
+    """
+    A low-precision version of the default LayerNorm.
+    """
+
+    rms = "rms"
+    """
+    An RMSNorm implementation. When using ``torch.compile`` this is
+    probably the fastest implementation.
+    """
+
+    gemma_rms = "gemma_rms"
+    """
+    An RMSNorm implementation by gemmma. When using ``torch.compile`` this is
+    probably the fastest implementation.
+    """
+
+    amd_compatible = "amd_compatible"
+    """
+    LayerNorm implemented manually to work around an issue with ROCm.
+    """
+
+
+class ActivationType(StrEnum):
+    gelu = "gelu"
+    relu = "relu"
+    silu = "silu"
+    swiglu = "swiglu"
+
+
+class BlockType(StrEnum):
+    sequential = "sequential"
+    parallel = "parallel"
+
+    llama = "llama"
+    """
+    A block similar to the sequential block with slightly different
+    implementations of operations like attention to imitate the behavior of Llama.
+    """
+
+
+class InitFnType(StrEnum):
+    mitchell = "mitchell"
+    """
+    The strategy suggested to us by Mitchell Wortsman from UW.
+    This uses a truncated normal distribution with an adaptive standard deviation that depends
+    on the size of the weights as well as the depth of the layer.
+    """
+
+    normal = "normal"
+    """
+    All weights are initialized from the same normal distribution.
+    """
+
+    kaiming_normal = "kaiming_normal"
+    """
+    All weights are initialized with the Kaiming method from a normal distribution.
+    Note this currently won't work with FSDP.
+    """
+
+    fan_in = "fan_in"
+    """
+    "Fan-in variance scaling", i.e. normal with a standard deviation of ``1/sqrt(d_in)`` where ``d_in``
+    is the input dimensionality of the kernel.
+    """
+
+    full_megatron = "full_megatron"
+    """
+    This is what metaseq calls "full megatron init". It is the init used for Llama 2.
+    """
+
+
+@dataclass
+class ModelConfig():
+    """
+    LLaDA (model) configuration.
+    """
+
+    # Note that the defaults for these attributes are equivalent to the base GPT2 model.
+
+    d_model: int = 768
+    """
+    The hidden size of the model.
+    """
+
+    n_heads: int = 12
+    """
+    The number of self-attention heads.
+    """
+
+    n_kv_heads: Optional[int] = None
+    """
+    The number of heads to use for keys and values. Defaults to `n_heads`.
+    Set this to ``None`` or ``n_heads`` for normal multi-head attention.
+    Set this to 1 for multi-query attention.
+    Set it to some in-between value for Llama2-style grouped query attention.
+    """
+
+    n_layers: int = 12
+    """
+    The number of layers/blocks.
+    """
+
+    mlp_ratio: int = 4
+    """
+    The ratio of the inner MLP dimensionality to ``d_model``.
+    This is only used when ``mlp_hidden_size`` is not set.
+    """
+
+    mlp_hidden_size: Optional[int] = None
+    """
+    Set the exact hidden size for the MLP. Otherwise the inner MLP hidden size will be set to `mlp_ratio * d_model`.
+    """
+
+    activation_type: ActivationType = ActivationType.swiglu
+    """
+    The activation function to use within the MLP layers.
+    """
+
+    block_type: BlockType = BlockType.sequential
+    """
+    The transformer block implementation.
+    """
+
+    block_group_size: int = 1
+    """
+    The number of blocks to group together into a single parent block.
+    This has no affect on the number of parameters in the model and is only used to wrap groups
+    of blocks together with a single FSDP wrapper during training.
+    """
+
+    alibi: bool = False
+    """
+    If ``True``, use ALiBi embeddings. Mutually exclusive with ``rope``.
+    """
+
+    alibi_bias_max: float = 8.0
+    """
+    Maximum absolute value of ALiBi bias.
+    """
+
+    rope: bool = False
+    """
+    Use rotary positional embeddings (RoPE). Mutually exclusive with ``alibi``.
+    """
+
+    rope_full_precision: bool = True
+    """
+    If ``True``, apply RoPE embeddings at full precision regardless of the input type. Otherwise,
+    apply RoPE at the precision of the input.
+    """
+
+    flash_attention: bool = False
+    """
+    If ``True``, use ``FlashAttention``.
+    """
+
+    attention_dropout: float = 0.1
+    """
+    The dropout probability within the attention modules.
+    """
+
+    multi_query_attention: Optional[bool] = None
+    """
+    Use the Multi-Query formulation of attention used in PaLM. This reduces the number of parameters
+    and is more efficient during inference.
+    """
+
+    attention_layer_norm: bool = False
+    """
+    Apply layer norm to the keys and queries within the attention mechanism.
+    This can help stabilize training.
+    """
+
+    residual_dropout: float = 0.1
+    """
+    The dropout probability for the MLP and attention output within each block.
+    """
+
+    embedding_dropout: float = 0.1
+    """
+    The dropout probability for embeddings.
+    """
+
+    input_emb_norm: bool = False
+    """
+    An input hidden_states norm implementation by gemmma.
+    """
+
+    layer_norm_type: LayerNormType = LayerNormType.default
+    """
+    The layernorm implementation to use.
+    """
+
+    layer_norm_with_affine: bool = True
+    """
+    Whether to include bias and weight parameters for the layer norms.
+    This only affects layer norms that are immediately followed by a linear layer in the forward pass,
+    so everything except QK-norms. To turn off affines for QK norms as well, set :attr:`attention_layer_norm_with_affine`
+    to ``False``.
+    """
+
+    rms_norm_eps: float = 1e-05
+    """
+    The rms layernorm eps param.
+    """
+
+    attention_layer_norm_with_affine: bool = True
+    """
+    Toggle affine transform for the QK norms.
+    """
+
+    max_sequence_length: int = 1024
+    """
+    The maximum input sequence length supported by the model.
+    """
+
+    rope_theta: float = 10000.0
+    """
+    The rope base param.
+    """
+
+    include_qkv_bias: Optional[bool] = False
+    """
+    Whether or not to include bias parameters in qkv linear layers.
+    """
+
+    include_bias: bool = False
+    """
+    Whether or not to include bias parameters in linear layers.
+    In PaLM, they got rid of all bias terms because they found that large
+    models tend to have near 0 bias terms anyway.
+    """
+
+    bias_for_layer_norm: Optional[bool] = None
+    """
+    Whether or not to include bias parameters in layer norm.
+    This is separate from the include_bias parameter, because of a ROCm crash when biases are disabled in
+    layer norm.
+    When this is None (the default), it inherits the setting from include_bias.
+    """
+
+    scale_logits: bool = False
+    """
+    If ``True``, scale the output logits by ``1 / sqrt(d_model)``.
+    """
+
+    vocab_size: int = 50257
+    """
+    Vocabulary size of the model.
+    """
+
+    embedding_size: Optional[int] = 50304
+    """
+    The number of embeddings, i.e. the number of tokens. If set to ``None`` it will default
+    to ``vocab_size``. If ``vocab_size`` is not a multiple of 128, setting this to the
+    next multiple of 128 that's greater than ``vocab_size`` can improve throughput
+    substantially.
+    """
+
+    weight_tying: bool = True
+    """
+    Whether to tie output linear weights to the input embedding.
+    """
+
+    eos_token_id: int = 50256
+    """
+    The ID of the end-of-sentence special token.
+    """
+
+    pad_token_id: int = 50256
+    """
+    The ID of the token to use for padding. Defaults to the ID of the EOS token.
+    """
+
+    mask_token_id: Optional[int] = 50256
+    """
+    The ID of the token to use for mask token. Defaults to the ID of the EOS token.
+    """
+
+    init_device: Optional[str] = None
+    """
+    The torch device to use when initializing the model parameters, e.g. "cpu", "cuda:0", "meta".
+    """
+
+    init_fn: InitFnType = InitFnType.normal
+    """
+    The weight initialization strategy.
+    """
+
+    init_std: float = 0.02
+    """
+    The standard deviation to use when initializing weights with a "fixed distribution" ``init_fn``, such
+    as "normal".
+    """
+
+    init_cutoff_factor: Optional[float] = None
+    """
+    A positive factor used to scale the cutoff values when initializing weights with a "fixed distribution" ``init_fn``, such
+    as "normal". Setting this to None means values are not cutoff.
+    """
+
+    precision: Optional[str] = None
+    """
+    Precision used to train/evaluate with. You shouldn't set this directly.
+    See :data:`TrainConfig.precision` instead.
+    """
+
+    @property
+    def effective_n_kv_heads(self) -> int:
+        if self.n_kv_heads is None:
+            if self.multi_query_attention is True:
+                return 1
+            else:
+                return self.n_heads
+        else:
+            if self.multi_query_attention is None:
+                return self.n_kv_heads
+            if self.multi_query_attention:
+                n_kv_heads_should_be = 1
+            else:
+                n_kv_heads_should_be = self.n_heads
+            if self.n_kv_heads == n_kv_heads_should_be:
+                return n_kv_heads_should_be
+            else:
+                raise Exception(
+                    "You can't set `multi_query_attention` and `n_kv_heads` at the same time."
+                )
+
+class ActivationCheckpointingStrategy(StrEnum):
+    whole_layer = "whole_layer"
+    """
+    Checkpoint every transformer layer.
+    """
+
+    one_in_two = "one_in_two"
+    """
+    Checkpoint one in two transformer layers.
+    """
+
+    one_in_three = "one_in_three"
+    """
+    Checkpoint one in three transformer layers.
+    """
+
+    one_in_four = "one_in_four"
+    """
+    Checkpoint one in four transformer layers.
+    """
+    
+    two_in_three = "two_in_three"
+    """
+    Checkpoint two out of every three transformer layers.
+    """
+
+    three_in_four = "three_in_four"
+    """
+    Checkpoint three out of four of every transformer layers.
+    """
+
+    four_in_five = "four_in_five"
+    """
+    Checkpoint four out of five of every transformer layers.
+    """
+
+    nine_in_ten = "nine_in_ten"
+    """
+    Checkpoint nine out of ten of every transformer layers.
+    """
+
+    fine_grained = "fine_grained"
+    """
+    Focus checkpointing on where it is cheap to recompute and saves most memory.
+    """
+
+
+class LLaDAConfig(PretrainedConfig):
+    model_type = "llada"
+    keys_to_ignore_at_inference = ["past_key_values"]  # TODO: confirm
+
+    def __init__(self, use_cache: bool = False, **kwargs):
+        model_config = ModelConfig()
+        all_kwargs = model_config.__dict__
+        all_kwargs.update(kwargs)
+        all_kwargs.update({"use_cache": use_cache})
+        all_kwargs.update(
+            {
+                "architectures": all_kwargs.get("architectures", ["LLaDAModelLM"])
+            }
+        )
+        super().__init__(**all_kwargs)
+
+    @property
+    def num_attention_heads(self):
+        return self.n_heads
+
+    @property
+    def num_hidden_layers(self):
+        return self.n_layers
+
+    @property
+    def hidden_size(self):
+        return self.d_model
+
+
+# Register the config class so that it is available for transformer pipelines, auto-loading etc.
+AutoConfig.register("llada", LLaDAConfig)

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 126080,
+  "eos_token_id": 126081,
+  "transformers_version": "4.49.0"
+}

model-00001-of-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2ba57d9e97151bf3d3f2e7c8aae4484a6b62c1bf7c123014a665b9adb681546e
+size 4957818648

model-00002-of-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6b562142d6c8181b621b9e9ecf531b9f50010b20a685900a313d710d7af2fdb0
+size 4966225704

model-00003-of-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:41f274c80f9b4f03e741a885c6e428c699e8c586ea0e20d373faec49c04e1297
+size 4832040992

model-00004-of-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e8f5a83bcf4ed934892b8013edf267169b4ed91b513d6fc5bde7562871ec7f1c
+size 4832007792

model-00005-of-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9b118ea76d89845520f7eda51e6ec30e4da9863ec5edbc8c560e42a0708ef477
+size 4966258728

model-00006-of-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:43be668a8ccc838990c4357fad3a486a00ea2bebb10bf0c22a11ba2436b47bb8
+size 4966258752

model-00007-of-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:608589c5a0461e0f3b33fd28be4c21ce978d380a29353f9aefdc35fe3657b916
+size 2541748728

model.safetensors.index.json

@@ -0,0 +1,298 @@
+{
+  "metadata": {
+    "total_size": 32062324736
+  },
+  "weight_map": {
+    "model.transformer.blocks.0.attn_norm.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.0.attn_out.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.0.ff_norm.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.0.ff_out.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.0.ff_proj.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.0.k_proj.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.0.q_proj.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.0.up_proj.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.0.v_proj.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.1.attn_norm.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.1.attn_out.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.1.ff_norm.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.1.ff_out.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.1.ff_proj.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.1.k_proj.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.1.q_proj.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.1.up_proj.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.1.v_proj.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.10.attn_norm.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.10.attn_out.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.10.ff_norm.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.10.ff_out.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.10.ff_proj.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.10.k_proj.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.10.q_proj.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.10.up_proj.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.10.v_proj.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.11.attn_norm.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.11.attn_out.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.11.ff_norm.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.11.ff_out.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.11.ff_proj.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.11.k_proj.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.11.q_proj.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.11.up_proj.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.11.v_proj.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.12.attn_norm.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.12.attn_out.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.12.ff_norm.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.12.ff_out.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.12.ff_proj.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.12.k_proj.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.12.q_proj.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.12.up_proj.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.12.v_proj.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.13.attn_norm.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.13.attn_out.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.13.ff_norm.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.13.ff_out.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.13.ff_proj.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.13.k_proj.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.13.q_proj.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.13.up_proj.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.13.v_proj.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.14.attn_norm.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.14.attn_out.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.14.ff_norm.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.14.ff_out.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.14.ff_proj.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.14.k_proj.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.14.q_proj.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.14.up_proj.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.14.v_proj.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.15.attn_norm.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.15.attn_out.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.15.ff_norm.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.15.ff_out.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.15.ff_proj.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.15.k_proj.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.15.q_proj.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.15.up_proj.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.15.v_proj.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.16.attn_norm.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.16.attn_out.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.16.ff_norm.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.16.ff_out.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.16.ff_proj.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.16.k_proj.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.16.q_proj.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.16.up_proj.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.16.v_proj.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.17.attn_norm.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.17.attn_out.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.17.ff_norm.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.17.ff_out.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.17.ff_proj.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.17.k_proj.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.17.q_proj.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.17.up_proj.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.17.v_proj.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.18.attn_norm.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.18.attn_out.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.18.ff_norm.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.18.ff_out.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.18.ff_proj.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.18.k_proj.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.18.q_proj.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.18.up_proj.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.18.v_proj.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.19.attn_norm.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.19.attn_out.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.19.ff_norm.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.19.ff_out.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.19.ff_proj.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.19.k_proj.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.19.q_proj.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.19.up_proj.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.19.v_proj.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.2.attn_norm.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.2.attn_out.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.2.ff_norm.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.2.ff_out.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.2.ff_proj.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.2.k_proj.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.2.q_proj.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.2.up_proj.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.2.v_proj.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.20.attn_norm.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.20.attn_out.weight": "model-00004-of-00007.safetensors",
+    "model.transformer.blocks.20.ff_norm.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.20.ff_out.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.20.ff_proj.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.20.k_proj.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.20.q_proj.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.20.up_proj.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.20.v_proj.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.21.attn_norm.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.21.attn_out.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.21.ff_norm.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.21.ff_out.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.21.ff_proj.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.21.k_proj.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.21.q_proj.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.21.up_proj.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.21.v_proj.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.22.attn_norm.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.22.attn_out.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.22.ff_norm.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.22.ff_out.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.22.ff_proj.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.22.k_proj.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.22.q_proj.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.22.up_proj.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.22.v_proj.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.23.attn_norm.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.23.attn_out.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.23.ff_norm.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.23.ff_out.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.23.ff_proj.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.23.k_proj.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.23.q_proj.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.23.up_proj.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.23.v_proj.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.24.attn_norm.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.24.attn_out.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.24.ff_norm.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.24.ff_out.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.24.ff_proj.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.24.k_proj.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.24.q_proj.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.24.up_proj.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.24.v_proj.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.25.attn_norm.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.25.attn_out.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.25.ff_norm.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.25.ff_out.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.25.ff_proj.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.25.k_proj.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.25.q_proj.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.25.up_proj.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.25.v_proj.weight": "model-00005-of-00007.safetensors",
+    "model.transformer.blocks.26.attn_norm.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.26.attn_out.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.26.ff_norm.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.26.ff_out.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.26.ff_proj.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.26.k_proj.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.26.q_proj.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.26.up_proj.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.26.v_proj.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.27.attn_norm.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.27.attn_out.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.27.ff_norm.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.27.ff_out.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.27.ff_proj.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.27.k_proj.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.27.q_proj.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.27.up_proj.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.27.v_proj.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.28.attn_norm.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.28.attn_out.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.28.ff_norm.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.28.ff_out.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.28.ff_proj.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.28.k_proj.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.28.q_proj.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.28.up_proj.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.28.v_proj.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.29.attn_norm.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.29.attn_out.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.29.ff_norm.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.29.ff_out.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.29.ff_proj.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.29.k_proj.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.29.q_proj.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.29.up_proj.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.29.v_proj.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.3.attn_norm.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.3.attn_out.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.3.ff_norm.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.3.ff_out.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.blocks.3.ff_proj.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.3.k_proj.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.3.q_proj.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.3.up_proj.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.3.v_proj.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.30.attn_norm.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.30.attn_out.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.30.ff_norm.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.30.ff_out.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.30.ff_proj.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.30.k_proj.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.30.q_proj.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.30.up_proj.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.30.v_proj.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.31.attn_norm.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.31.attn_out.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.31.ff_norm.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.31.ff_out.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.31.ff_proj.weight": "model-00007-of-00007.safetensors",
+    "model.transformer.blocks.31.k_proj.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.31.q_proj.weight": "model-00006-of-00007.safetensors",
+    "model.transformer.blocks.31.up_proj.weight": "model-00007-of-00007.safetensors",
+    "model.transformer.blocks.31.v_proj.weight": "model-00007-of-00007.safetensors",
+    "model.transformer.blocks.4.attn_norm.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.4.attn_out.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.4.ff_norm.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.4.ff_out.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.4.ff_proj.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.4.k_proj.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.4.q_proj.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.4.up_proj.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.4.v_proj.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.5.attn_norm.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.5.attn_out.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.5.ff_norm.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.5.ff_out.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.5.ff_proj.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.5.k_proj.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.5.q_proj.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.5.up_proj.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.5.v_proj.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.6.attn_norm.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.6.attn_out.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.6.ff_norm.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.6.ff_out.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.6.ff_proj.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.6.k_proj.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.6.q_proj.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.6.up_proj.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.6.v_proj.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.7.attn_norm.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.7.attn_out.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.7.ff_norm.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.7.ff_out.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.7.ff_proj.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.7.k_proj.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.7.q_proj.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.7.up_proj.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.7.v_proj.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.8.attn_norm.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.8.attn_out.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.8.ff_norm.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.8.ff_out.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.8.ff_proj.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.8.k_proj.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.8.q_proj.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.8.up_proj.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.8.v_proj.weight": "model-00002-of-00007.safetensors",
+    "model.transformer.blocks.9.attn_norm.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.9.attn_out.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.9.ff_norm.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.9.ff_out.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.9.ff_proj.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.9.k_proj.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.9.q_proj.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.9.up_proj.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.blocks.9.v_proj.weight": "model-00003-of-00007.safetensors",
+    "model.transformer.ff_out.weight": "model-00007-of-00007.safetensors",
+    "model.transformer.ln_f.weight": "model-00001-of-00007.safetensors",
+    "model.transformer.wte.weight": "model-00001-of-00007.safetensors"
+  }
+}