Graph-free API whole-call pilot
Contents
Each API runs on its own appropriate random binary SCM. These are different estimation problems, so the table is not a head-to-head ranking of interchangeable methods.
Configuration: n=1200, seeds=[0, 1, 2, 3, 4], XGBoost rounds=30, CPU nthread=1. Mechanisms are fixed per graph; only observational samples and splits vary.
Time includes the entire public API call: input checks, expression/program construction, nuisance fitting and score evaluation. Data generation and graph setup/identification are recorded separately. Every API has one excluded warm-up. Recorded trials execute sequentially with no concurrent tests.
| API | Graph | Exact mean | Bias | RMSE | MAE | Median call seconds | Failures |
|---|---|---|---|---|---|---|---|
| estimate_bd | BD | 0.506880 | -0.007478 | 0.024279 | 0.018542 | 3.709566 | 0/5 |
| estimate_sbd | SBD | 0.490578 | -0.009034 | 0.066171 | 0.060007 | 4.102618 | 0/5 |
| estimate_fd | FD | 0.366866 | 0.016239 | 0.020983 | 0.016239 | 5.780009 | 0/5 |
| estimate_uca | MC-only | 0.496036 | 0.018974 | 0.042063 | 0.038043 | 13.480975 | 0/5 |
Bias/RMSE/MAE use successful trials, with the full failure denominator shown. Call time uses all attempted calls, including explicit failures.
| API | Setup/ID once (s) | Median generation (s) | Stages | Held-out rows |
|---|---|---|---|---|
| estimate_bd | 0.000002 | 0.001771 | 2 | 480 |
| estimate_sbd | 0.000003 | 0.001328 | 2 | 480 |
| estimate_fd | 0.000003 | 0.003158 | 3 | 513 |
| estimate_uca | 0.182079 | 0.007323 | 4 | 532 |
BD uses X adjustment. SBD sets B and A, with L measured between them. FD uses mediator M with hidden A–Y confounding. General UCA uses the embedded MC-only graph and a graph-free supplied expression; its FMC/MC setup is excluded from API time.
All targets are interventional means under the listed actions set to one. Exact truth enumerates the complete SCM, including latent variables, omitting intervened action mechanisms. Learners see observed data only.
Environment: Python 3.11.14; macOS-15.6.1-arm64-arm-64bit; causalengine 0.1.0a13. Source baseline 0dcd28d49149ee29796231f33d669810394e6b40 plus module/script hashes in raw data.
This five-seed pilot measures point accuracy and machine/session wall time. It establishes no general speed ordering, causal interval coverage or conditional-calibration guarantee. ATE/CATE independent heterogeneous-truth tests are separate from this interventional-mean benchmark. Held-out pseudo-outcome MSE in a CATE result is not CATE truth error.
Raw mechanisms, observations/split seeds, per-call outcomes and source hashes: JSON results. Reproduce with python tutorials/benchmark_estimators.py --seeds 5 --output /tmp/api-results.json --report /tmp/api-report.md; --seeds is a count beginning at zero. Use the exact installed candidate source hashes, not only its version.