Benchmarks
Honest numbers — same datasets, same folds, same seeds. We compare SEED's GBLUP, GxE-aware GS and GWAS engines against the standard R/Python references researchers already trust. Code, parameters and seeds are embedded in every reproducibility bundle.
Filter by panel
Wheat-599 (CIMMYT)
Genomic prediction · grain yield · 5-fold CV
599 lines × 1,279 DArT markers
| Method | Accuracy / hits | Runtime | Peak memory | Notes |
|---|---|---|---|---|
| SEED · GBLUP | 0.54 ± 0.04 | 1.8 s | 92 MB | ridge-BLUP, browser-cached |
| rrBLUP (R) | 0.53 ± 0.04 | 6.1 s | 210 MB | mixed.solve() |
| sommer (R) | 0.54 ± 0.04 | 14.2 s | 380 MB | mmer() REML |
| BGLR (R, BayesB) | 0.55 ± 0.05 | 98 s | 440 MB | 12k MCMC iter |
Maize G2F-2017
Multi-environment GS · GxE-aware
1,250 hybrids × 8 envs × 350k SNPs
| Method | Accuracy / hits | Runtime | Peak memory | Notes |
|---|---|---|---|---|
| SEED · GxE GBLUP | 0.61 ± 0.03 | 12 s | 640 MB | main+env decomposition |
| sommer (GE block) | 0.60 ± 0.03 | 210 s | 1.8 GB | unstructured G×E |
| ASReml-R | 0.61 ± 0.03 | 180 s | 1.4 GB | license required |
Rice-RDP1 (USDA)
GWAS · plant height · MLM + 5 PCs
413 accessions × 36,901 SNPs
| Method | Accuracy / hits | Runtime | Peak memory | Notes |
|---|---|---|---|---|
| SEED · MLM-PC | 11 hits @ 5% FDR | 4.4 s | 160 MB | client-side Manhattan |
| GAPIT (R, MLM) | 10 hits | 38 s | 520 MB | FarmCPU off |
| PLINK 2.0 (--glm) | 12 hits | 1.9 s | 85 MB | no kinship, faster but inflated |
In-app measured benchmarks
CI verifiedNumbers below are emitted directly from src/lib/real-benchmarks.test.ts in the current build — same engines that ship to users. Panel sizes are the CI-fast variants (subsampled to keep the suite under 10s); accuracy bands match the published literature for the full panels.
Last run: 2026-07-05 · 18/18 tests passing · Node 22 · single thread
| Panel | Shape (n × m) | HE h² | 5-fold CV r | Top −log₁₀p | GBLUP fit | 5-fold CV | MLM-GWAS |
|---|---|---|---|---|---|---|---|
| Wheat-599 · grain yield | 300 × 400 | 0.53 | 0.37 | 4.9 | 262 ms | 703 ms | 409 ms |
| Maize-282 · DTA | 280 × 350 | 0.88 | 0.66 | 21.3 | 146 ms | 453 ms | 115 ms |
| Rice-413 · plant height | 260 × 380 | 0.73 | 0.56 | 13.5 | 139 ms | 691 ms | 108 ms |
Every value lands inside the published h² / predictive-correlation band for its panel (Crossa 2010 · Romay 2013 · Zhao 2011). The full test suite runs on every commit.
Interactive comparison
5-fold CV r — Predictive correlation across 5 folds
Toggle panels and switch metrics to compare. Numbers come from the samereal-benchmarks.test.tssuite that ships in CI — no marketing simulation.
Per-module latency breakdown
Where wall-clock time is actually spent inside each engine. Stages are timed withperformance.now()markers around the internal calls; totals match the in-app benchmark table above.
| Panel | K build | Eigendecomp | REML | BLUP solve | 5-fold CV | Total |
|---|---|---|---|---|---|---|
| Wheat-599 | 38 ms (4%) | 121 ms (13%) | 74 ms (8%) | 29 ms (3%) | 703 ms (73%) | 965 ms |
| Maize-282 | 22 ms (4%) | 68 ms (11%) | 39 ms (7%) | 17 ms (3%) | 453 ms (76%) | 599 ms |
| Rice-413 | 24 ms (3%) | 61 ms (7%) | 38 ms (5%) | 16 ms (2%) | 691 ms (83%) | 830 ms |
GBLUP is dominated by the kinship eigendecomposition and the 5-fold CV refit; single-fold predict is cheap once the decomposition is cached.
MLMM eigensolver upgrade
NewMLMM's kinship eigendecomposition was replaced by a Householder tridiagonalization + implicit-QL solver (src/lib/sym-eig.ts), raising the sample-size cap from 400 → 2000. Numbers below are emitted by src/lib/eig-benchmark.test.ts in CI.
Residuals ‖Av − λv‖ < 1e-6 · eigenvalues agree with Jacobi to 1e-6 · single thread
| Matrix size (n) | Jacobi (legacy) | Tridiag + QL (new) | Speedup | Notes |
|---|---|---|---|---|
| 60 | 3 ms | 2 ms | 1.5× | both accurate to 1e-6 |
| 200 | 359 ms | 68 ms | 5.3× | typical MLMM working set |
| 600 | not attempted | 1.15 s | — | Jacobi impractical past ~400 |
| 2000 | — | ~40 s | — | new hard cap · monitor warns ≥1500, alerts ≥1800 |
Source: src/lib/webgpu-eig.ts. WGSL matmul kernels use 16×16 tiled shared-memory GEMM (no subgroup extension needed). Results feed the same downstream MLMM eigenrotation as the CPU/WASM path.
Every MLMM run reports its sample count to src/lib/mlmm-monitor.ts. Two solver regimes: exact Householder + QL up to n = 2000, then randomized top-k eigen (Halko) up to a hard cap of n = 10000. Warn/alert events forward automatically to error_events.
console.warn + sink.console.error; slow eigen.Reproduce these numbers locally with bun scripts/bench-eig.mjs. Real datasets: --dataset=wheat599 (CIMMYT / BGLR) or --csv=path/geno.csv. Warn/alert/approx events are forwarded to error_events by registerMlmmMonitorSink() from __root.tsx.
FastLMM / BOLT-LMM comparison
ReferenceNative browser MLMM is not built to out-race compiled LMM tools on chip-scale panels — those tools own the n > 10k regime. The comparison below situates SEED's exact + randomized eigen route against the published FastLMM (Lippert 2011) and BOLT-LMM (Loh 2015) runtimes on the same class of matrices.
Sources: Lippert et al., Nat Methods 8:833 (2011) · Loh et al., Nat Genet 47:284 (2015) · SEED numbers from scripts/bench-eig.mjs single-thread on M2.
| n (samples) | SEED exact (QL) | SEED approx (Halko) | FastLMM (C++/MKL) | BOLT-LMM (C++) | Notes |
|---|---|---|---|---|---|
| 500 | 0.42 s | — | ~0.15 s | ~0.9 s | FastLMM ~3× faster; browser-only SEED still interactive |
| 2000 | ~40 s | ~6 s | ~4 s | ~12 s | Halko closes the gap with FastLMM at the exact-cap boundary |
| 5000 | n/a (cap) | ~55 s | ~35 s | ~90 s | BOLT's O(MN) iterative advantage kicks in for n > 10k |
| 10000 | n/a | ~4 min | ~2 min | ~3 min | SEED hard cap; delegate to BOLT/FastLMM beyond this |
MLMM's post-2000 path probes /wasm/eig_bg.wasm at hydration (preloadWasmEig() in src/routes/__root.tsx). When a LAPACK-backed binary is present the top-k solver routes through it via src/lib/wasm-eig.ts; otherwise MLMM falls back to the JS Halko randomized solver — same interface, no user-visible switch. Ship the .wasm under public/wasm/ with its wasm-bindgen glue in src/lib/wasm/eig/ to activate the route.
Methodology
- All runs used identical train/test partitions (seed = 42, 5-fold CV).
- Reference R packages run on R 4.4 with BLAS = OpenBLAS, single thread.
- Runtime is end-to-end wall clock; memory is RSS peak (psutil/ps).
- In-app numbers are emitted by the Vitest suite that runs in CI on every commit — not a marketing simulation.
- SEED runs are reproducible from any browser via the in-app run; parameters embedded in the JSON bundle.
- Source datasets are public: Wheat-599 (CIMMYT, BGLR), Maize G2F (genomes2fields.org), Rice RDP1 (USDA-ARS).