msluszniak commited on
Commit
57fbfd8
·
verified ·
1 Parent(s): 5215404

Remove MLX artifacts: slower than Core ML on device

Browse files

MLX measured slower than Core ML on an iPhone 16 for this model, at matched fp32 precision and against the shipped Core ML fp16 build. These exporters only ever declared an MLX fp32 variant, so fp32-vs-fp32 is the fair and conservative comparison, and MLX still loses it.

The .pte and config.json are removed; each mlx/ dir keeps a NOTES.md with this model's own measurements. The exporter is retained in export-scripts so this can be re-tested against a future ExecuTorch/MLX release.

MLX for these models was only ever published on main / v0.10.0; no tag at v0.9.0 or below carried it. v0.9.0 and below are untouched.

s/mlx/NOTES.md ADDED
@@ -0,0 +1,51 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # MLX is not shipped for this model
2
+
3
+ The MLX artifacts for this model were removed deliberately. The exporter in
4
+ `export-scripts` is kept so the decision can be re-tested against a future
5
+ ExecuTorch/MLX release, but no `.pte` or `config.json` is published here.
6
+
7
+ ## Why
8
+
9
+ Measured on device (iPhone 16, ExecuTorch 1.4.1, Release build, 3 warmup runs +
10
+ 15 timed runs, median), on fast-sam (s):
11
+
12
+ | backend | precision | median latency |
13
+ |---|---|---|
14
+ | Core ML | fp32 (precision matched) | 40.5 ms |
15
+ | MLX | fp32 (only precision offered) | 77.6 ms |
16
+ | Core ML | fp16 (shipped default) | **9.1 ms** |
17
+
18
+ MLX is **1.9x slower than Core ML at matched precision**, and **8.5x slower than
19
+ the Core ML build that actually ships**.
20
+
21
+ These exporters only ever declared an MLX fp32 variant, so fp32-vs-fp32 is the
22
+ fair comparison and it is deliberately the conservative one: MLX still loses it.
23
+
24
+ ## Context
25
+
26
+ MLX was measured behind Core ML on every convolutional vision model tested. The
27
+ margin varies widely with architecture, so each model carries its own number
28
+ rather than a blanket figure.
29
+
30
+ Two things compound the gap. MLX weight quantization cannot shrink a conv model:
31
+ `quantize_model_` only rewrites `nn.Linear` and embedding modules, and
32
+ EfficientNet-V2-S holds just 6.0% of its parameters in its single `nn.Linear`,
33
+ so 4-bit quantization buys about 5% file size. Core ML by contrast reaches the
34
+ convolutions.
35
+
36
+ ## Scope
37
+
38
+ This concerns convolution-dominated vision models. It does **not** generalize:
39
+
40
+ - `rfdetr-nano` (conv + transformer) measured only **1.24x** behind Core ML.
41
+ - The `lfm2.5` text encoders measured **2.7-3.1x faster on MLX** than XNNPACK.
42
+
43
+ MLX suits matmul-heavy graphs. The right backend depends on model class, so this
44
+ file is not a statement about MLX in general.
45
+
46
+ ## Availability history
47
+
48
+ MLX for this model was only ever published on `main` / `v0.10.0`. No earlier tag
49
+ (`v0.9.0` and below) carried an MLX artifact for it, so there is no pinned
50
+ revision from which these files can still be fetched. They remain recoverable
51
+ from this repository's git history.
s/mlx/config.json DELETED
@@ -1,62 +0,0 @@
1
- {
2
- "$schema": "https://huggingface.co/software-mansion/react-native-executorch-spec/resolve/main/config.schema.json",
3
- "model": "fast_sam_s",
4
- "family": "fastsam",
5
- "capabilities": [
6
- "instance-segmentation"
7
- ],
8
- "backend": "mlx",
9
- "license": "agpl-3.0",
10
- "variants": [
11
- {
12
- "file": "fast_sam_s_mlx_fp32.pte",
13
- "precision": "fp32",
14
- "quantized": false,
15
- "default": true,
16
- "methods": {
17
- "forward": {
18
- "inputs": [
19
- {
20
- "shape": [
21
- 1,
22
- 3,
23
- 640,
24
- 640
25
- ],
26
- "dtype": "float32"
27
- }
28
- ],
29
- "outputs": [
30
- {
31
- "shape": [
32
- 300,
33
- 4
34
- ],
35
- "dtype": "float32"
36
- },
37
- {
38
- "shape": [
39
- 300
40
- ],
41
- "dtype": "float32"
42
- },
43
- {
44
- "shape": [
45
- 300
46
- ],
47
- "dtype": "float32"
48
- },
49
- {
50
- "shape": [
51
- 300,
52
- 160,
53
- 160
54
- ],
55
- "dtype": "float32"
56
- }
57
- ]
58
- }
59
- }
60
- }
61
- ]
62
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
s/mlx/fast_sam_s_mlx_fp32.pte DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:0a47cd39e5e1cb0e7202c7e1f893a3ad819c3199f4a6dbe294a8881361340628
3
- size 47499520
 
 
 
 
x/mlx/NOTES.md ADDED
@@ -0,0 +1,51 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # MLX is not shipped for this model
2
+
3
+ The MLX artifacts for this model were removed deliberately. The exporter in
4
+ `export-scripts` is kept so the decision can be re-tested against a future
5
+ ExecuTorch/MLX release, but no `.pte` or `config.json` is published here.
6
+
7
+ ## Why
8
+
9
+ Measured on device (iPhone 16, ExecuTorch 1.4.1, Release build, 3 warmup runs +
10
+ 15 timed runs, median), on fast-sam (s):
11
+
12
+ | backend | precision | median latency |
13
+ |---|---|---|
14
+ | Core ML | fp32 (precision matched) | 40.5 ms |
15
+ | MLX | fp32 (only precision offered) | 77.6 ms |
16
+ | Core ML | fp16 (shipped default) | **9.1 ms** |
17
+
18
+ MLX is **1.9x slower than Core ML at matched precision**, and **8.5x slower than
19
+ the Core ML build that actually ships**.
20
+
21
+ These exporters only ever declared an MLX fp32 variant, so fp32-vs-fp32 is the
22
+ fair comparison and it is deliberately the conservative one: MLX still loses it.
23
+
24
+ ## Context
25
+
26
+ MLX was measured behind Core ML on every convolutional vision model tested. The
27
+ margin varies widely with architecture, so each model carries its own number
28
+ rather than a blanket figure.
29
+
30
+ Two things compound the gap. MLX weight quantization cannot shrink a conv model:
31
+ `quantize_model_` only rewrites `nn.Linear` and embedding modules, and
32
+ EfficientNet-V2-S holds just 6.0% of its parameters in its single `nn.Linear`,
33
+ so 4-bit quantization buys about 5% file size. Core ML by contrast reaches the
34
+ convolutions.
35
+
36
+ ## Scope
37
+
38
+ This concerns convolution-dominated vision models. It does **not** generalize:
39
+
40
+ - `rfdetr-nano` (conv + transformer) measured only **1.24x** behind Core ML.
41
+ - The `lfm2.5` text encoders measured **2.7-3.1x faster on MLX** than XNNPACK.
42
+
43
+ MLX suits matmul-heavy graphs. The right backend depends on model class, so this
44
+ file is not a statement about MLX in general.
45
+
46
+ ## Availability history
47
+
48
+ MLX for this model was only ever published on `main` / `v0.10.0`. No earlier tag
49
+ (`v0.9.0` and below) carried an MLX artifact for it, so there is no pinned
50
+ revision from which these files can still be fetched. They remain recoverable
51
+ from this repository's git history.
x/mlx/config.json DELETED
@@ -1,62 +0,0 @@
1
- {
2
- "$schema": "https://huggingface.co/software-mansion/react-native-executorch-spec/resolve/main/config.schema.json",
3
- "model": "fast_sam_x",
4
- "family": "fastsam",
5
- "capabilities": [
6
- "instance-segmentation"
7
- ],
8
- "backend": "mlx",
9
- "license": "agpl-3.0",
10
- "variants": [
11
- {
12
- "file": "fast_sam_x_mlx_fp32.pte",
13
- "precision": "fp32",
14
- "quantized": false,
15
- "default": true,
16
- "methods": {
17
- "forward": {
18
- "inputs": [
19
- {
20
- "shape": [
21
- 1,
22
- 3,
23
- 640,
24
- 640
25
- ],
26
- "dtype": "float32"
27
- }
28
- ],
29
- "outputs": [
30
- {
31
- "shape": [
32
- 300,
33
- 4
34
- ],
35
- "dtype": "float32"
36
- },
37
- {
38
- "shape": [
39
- 300
40
- ],
41
- "dtype": "float32"
42
- },
43
- {
44
- "shape": [
45
- 300
46
- ],
47
- "dtype": "float32"
48
- },
49
- {
50
- "shape": [
51
- 300,
52
- 160,
53
- 160
54
- ],
55
- "dtype": "float32"
56
- }
57
- ]
58
- }
59
- }
60
- }
61
- ]
62
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
x/mlx/fast_sam_x_mlx_fp32.pte DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:113f9d393e357f11d0195e72e690d852aa3ba277829d525ab6dcffb9442c3294
3
- size 289517568