Remove MLX artifacts: slower than Core ML on device
Browse filesMLX measured slower than Core ML on an iPhone 16 for this model, at matched fp32 precision and against the shipped Core ML fp16 build. These exporters only ever declared an MLX fp32 variant, so fp32-vs-fp32 is the fair and conservative comparison, and MLX still loses it.
The .pte and config.json are removed; each mlx/ dir keeps a NOTES.md with this model's own measurements. The exporter is retained in export-scripts so this can be re-tested against a future ExecuTorch/MLX release.
MLX for these models was only ever published on main / v0.10.0; no tag at v0.9.0 or below carried it. v0.9.0 and below are untouched.
- s/mlx/NOTES.md +51 -0
- s/mlx/config.json +0 -62
- s/mlx/fast_sam_s_mlx_fp32.pte +0 -3
- x/mlx/NOTES.md +51 -0
- x/mlx/config.json +0 -62
- x/mlx/fast_sam_x_mlx_fp32.pte +0 -3
s/mlx/NOTES.md
ADDED
|
@@ -0,0 +1,51 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# MLX is not shipped for this model
|
| 2 |
+
|
| 3 |
+
The MLX artifacts for this model were removed deliberately. The exporter in
|
| 4 |
+
`export-scripts` is kept so the decision can be re-tested against a future
|
| 5 |
+
ExecuTorch/MLX release, but no `.pte` or `config.json` is published here.
|
| 6 |
+
|
| 7 |
+
## Why
|
| 8 |
+
|
| 9 |
+
Measured on device (iPhone 16, ExecuTorch 1.4.1, Release build, 3 warmup runs +
|
| 10 |
+
15 timed runs, median), on fast-sam (s):
|
| 11 |
+
|
| 12 |
+
| backend | precision | median latency |
|
| 13 |
+
|---|---|---|
|
| 14 |
+
| Core ML | fp32 (precision matched) | 40.5 ms |
|
| 15 |
+
| MLX | fp32 (only precision offered) | 77.6 ms |
|
| 16 |
+
| Core ML | fp16 (shipped default) | **9.1 ms** |
|
| 17 |
+
|
| 18 |
+
MLX is **1.9x slower than Core ML at matched precision**, and **8.5x slower than
|
| 19 |
+
the Core ML build that actually ships**.
|
| 20 |
+
|
| 21 |
+
These exporters only ever declared an MLX fp32 variant, so fp32-vs-fp32 is the
|
| 22 |
+
fair comparison and it is deliberately the conservative one: MLX still loses it.
|
| 23 |
+
|
| 24 |
+
## Context
|
| 25 |
+
|
| 26 |
+
MLX was measured behind Core ML on every convolutional vision model tested. The
|
| 27 |
+
margin varies widely with architecture, so each model carries its own number
|
| 28 |
+
rather than a blanket figure.
|
| 29 |
+
|
| 30 |
+
Two things compound the gap. MLX weight quantization cannot shrink a conv model:
|
| 31 |
+
`quantize_model_` only rewrites `nn.Linear` and embedding modules, and
|
| 32 |
+
EfficientNet-V2-S holds just 6.0% of its parameters in its single `nn.Linear`,
|
| 33 |
+
so 4-bit quantization buys about 5% file size. Core ML by contrast reaches the
|
| 34 |
+
convolutions.
|
| 35 |
+
|
| 36 |
+
## Scope
|
| 37 |
+
|
| 38 |
+
This concerns convolution-dominated vision models. It does **not** generalize:
|
| 39 |
+
|
| 40 |
+
- `rfdetr-nano` (conv + transformer) measured only **1.24x** behind Core ML.
|
| 41 |
+
- The `lfm2.5` text encoders measured **2.7-3.1x faster on MLX** than XNNPACK.
|
| 42 |
+
|
| 43 |
+
MLX suits matmul-heavy graphs. The right backend depends on model class, so this
|
| 44 |
+
file is not a statement about MLX in general.
|
| 45 |
+
|
| 46 |
+
## Availability history
|
| 47 |
+
|
| 48 |
+
MLX for this model was only ever published on `main` / `v0.10.0`. No earlier tag
|
| 49 |
+
(`v0.9.0` and below) carried an MLX artifact for it, so there is no pinned
|
| 50 |
+
revision from which these files can still be fetched. They remain recoverable
|
| 51 |
+
from this repository's git history.
|
s/mlx/config.json
DELETED
|
@@ -1,62 +0,0 @@
|
|
| 1 |
-
{
|
| 2 |
-
"$schema": "https://huggingface.co/software-mansion/react-native-executorch-spec/resolve/main/config.schema.json",
|
| 3 |
-
"model": "fast_sam_s",
|
| 4 |
-
"family": "fastsam",
|
| 5 |
-
"capabilities": [
|
| 6 |
-
"instance-segmentation"
|
| 7 |
-
],
|
| 8 |
-
"backend": "mlx",
|
| 9 |
-
"license": "agpl-3.0",
|
| 10 |
-
"variants": [
|
| 11 |
-
{
|
| 12 |
-
"file": "fast_sam_s_mlx_fp32.pte",
|
| 13 |
-
"precision": "fp32",
|
| 14 |
-
"quantized": false,
|
| 15 |
-
"default": true,
|
| 16 |
-
"methods": {
|
| 17 |
-
"forward": {
|
| 18 |
-
"inputs": [
|
| 19 |
-
{
|
| 20 |
-
"shape": [
|
| 21 |
-
1,
|
| 22 |
-
3,
|
| 23 |
-
640,
|
| 24 |
-
640
|
| 25 |
-
],
|
| 26 |
-
"dtype": "float32"
|
| 27 |
-
}
|
| 28 |
-
],
|
| 29 |
-
"outputs": [
|
| 30 |
-
{
|
| 31 |
-
"shape": [
|
| 32 |
-
300,
|
| 33 |
-
4
|
| 34 |
-
],
|
| 35 |
-
"dtype": "float32"
|
| 36 |
-
},
|
| 37 |
-
{
|
| 38 |
-
"shape": [
|
| 39 |
-
300
|
| 40 |
-
],
|
| 41 |
-
"dtype": "float32"
|
| 42 |
-
},
|
| 43 |
-
{
|
| 44 |
-
"shape": [
|
| 45 |
-
300
|
| 46 |
-
],
|
| 47 |
-
"dtype": "float32"
|
| 48 |
-
},
|
| 49 |
-
{
|
| 50 |
-
"shape": [
|
| 51 |
-
300,
|
| 52 |
-
160,
|
| 53 |
-
160
|
| 54 |
-
],
|
| 55 |
-
"dtype": "float32"
|
| 56 |
-
}
|
| 57 |
-
]
|
| 58 |
-
}
|
| 59 |
-
}
|
| 60 |
-
}
|
| 61 |
-
]
|
| 62 |
-
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
s/mlx/fast_sam_s_mlx_fp32.pte
DELETED
|
@@ -1,3 +0,0 @@
|
|
| 1 |
-
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:0a47cd39e5e1cb0e7202c7e1f893a3ad819c3199f4a6dbe294a8881361340628
|
| 3 |
-
size 47499520
|
|
|
|
|
|
|
|
|
|
|
|
x/mlx/NOTES.md
ADDED
|
@@ -0,0 +1,51 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# MLX is not shipped for this model
|
| 2 |
+
|
| 3 |
+
The MLX artifacts for this model were removed deliberately. The exporter in
|
| 4 |
+
`export-scripts` is kept so the decision can be re-tested against a future
|
| 5 |
+
ExecuTorch/MLX release, but no `.pte` or `config.json` is published here.
|
| 6 |
+
|
| 7 |
+
## Why
|
| 8 |
+
|
| 9 |
+
Measured on device (iPhone 16, ExecuTorch 1.4.1, Release build, 3 warmup runs +
|
| 10 |
+
15 timed runs, median), on fast-sam (s):
|
| 11 |
+
|
| 12 |
+
| backend | precision | median latency |
|
| 13 |
+
|---|---|---|
|
| 14 |
+
| Core ML | fp32 (precision matched) | 40.5 ms |
|
| 15 |
+
| MLX | fp32 (only precision offered) | 77.6 ms |
|
| 16 |
+
| Core ML | fp16 (shipped default) | **9.1 ms** |
|
| 17 |
+
|
| 18 |
+
MLX is **1.9x slower than Core ML at matched precision**, and **8.5x slower than
|
| 19 |
+
the Core ML build that actually ships**.
|
| 20 |
+
|
| 21 |
+
These exporters only ever declared an MLX fp32 variant, so fp32-vs-fp32 is the
|
| 22 |
+
fair comparison and it is deliberately the conservative one: MLX still loses it.
|
| 23 |
+
|
| 24 |
+
## Context
|
| 25 |
+
|
| 26 |
+
MLX was measured behind Core ML on every convolutional vision model tested. The
|
| 27 |
+
margin varies widely with architecture, so each model carries its own number
|
| 28 |
+
rather than a blanket figure.
|
| 29 |
+
|
| 30 |
+
Two things compound the gap. MLX weight quantization cannot shrink a conv model:
|
| 31 |
+
`quantize_model_` only rewrites `nn.Linear` and embedding modules, and
|
| 32 |
+
EfficientNet-V2-S holds just 6.0% of its parameters in its single `nn.Linear`,
|
| 33 |
+
so 4-bit quantization buys about 5% file size. Core ML by contrast reaches the
|
| 34 |
+
convolutions.
|
| 35 |
+
|
| 36 |
+
## Scope
|
| 37 |
+
|
| 38 |
+
This concerns convolution-dominated vision models. It does **not** generalize:
|
| 39 |
+
|
| 40 |
+
- `rfdetr-nano` (conv + transformer) measured only **1.24x** behind Core ML.
|
| 41 |
+
- The `lfm2.5` text encoders measured **2.7-3.1x faster on MLX** than XNNPACK.
|
| 42 |
+
|
| 43 |
+
MLX suits matmul-heavy graphs. The right backend depends on model class, so this
|
| 44 |
+
file is not a statement about MLX in general.
|
| 45 |
+
|
| 46 |
+
## Availability history
|
| 47 |
+
|
| 48 |
+
MLX for this model was only ever published on `main` / `v0.10.0`. No earlier tag
|
| 49 |
+
(`v0.9.0` and below) carried an MLX artifact for it, so there is no pinned
|
| 50 |
+
revision from which these files can still be fetched. They remain recoverable
|
| 51 |
+
from this repository's git history.
|
x/mlx/config.json
DELETED
|
@@ -1,62 +0,0 @@
|
|
| 1 |
-
{
|
| 2 |
-
"$schema": "https://huggingface.co/software-mansion/react-native-executorch-spec/resolve/main/config.schema.json",
|
| 3 |
-
"model": "fast_sam_x",
|
| 4 |
-
"family": "fastsam",
|
| 5 |
-
"capabilities": [
|
| 6 |
-
"instance-segmentation"
|
| 7 |
-
],
|
| 8 |
-
"backend": "mlx",
|
| 9 |
-
"license": "agpl-3.0",
|
| 10 |
-
"variants": [
|
| 11 |
-
{
|
| 12 |
-
"file": "fast_sam_x_mlx_fp32.pte",
|
| 13 |
-
"precision": "fp32",
|
| 14 |
-
"quantized": false,
|
| 15 |
-
"default": true,
|
| 16 |
-
"methods": {
|
| 17 |
-
"forward": {
|
| 18 |
-
"inputs": [
|
| 19 |
-
{
|
| 20 |
-
"shape": [
|
| 21 |
-
1,
|
| 22 |
-
3,
|
| 23 |
-
640,
|
| 24 |
-
640
|
| 25 |
-
],
|
| 26 |
-
"dtype": "float32"
|
| 27 |
-
}
|
| 28 |
-
],
|
| 29 |
-
"outputs": [
|
| 30 |
-
{
|
| 31 |
-
"shape": [
|
| 32 |
-
300,
|
| 33 |
-
4
|
| 34 |
-
],
|
| 35 |
-
"dtype": "float32"
|
| 36 |
-
},
|
| 37 |
-
{
|
| 38 |
-
"shape": [
|
| 39 |
-
300
|
| 40 |
-
],
|
| 41 |
-
"dtype": "float32"
|
| 42 |
-
},
|
| 43 |
-
{
|
| 44 |
-
"shape": [
|
| 45 |
-
300
|
| 46 |
-
],
|
| 47 |
-
"dtype": "float32"
|
| 48 |
-
},
|
| 49 |
-
{
|
| 50 |
-
"shape": [
|
| 51 |
-
300,
|
| 52 |
-
160,
|
| 53 |
-
160
|
| 54 |
-
],
|
| 55 |
-
"dtype": "float32"
|
| 56 |
-
}
|
| 57 |
-
]
|
| 58 |
-
}
|
| 59 |
-
}
|
| 60 |
-
}
|
| 61 |
-
]
|
| 62 |
-
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
x/mlx/fast_sam_x_mlx_fp32.pte
DELETED
|
@@ -1,3 +0,0 @@
|
|
| 1 |
-
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:113f9d393e357f11d0195e72e690d852aa3ba277829d525ab6dcffb9442c3294
|
| 3 |
-
size 289517568
|
|
|
|
|
|
|
|
|
|
|
|