Study 01 · complete findings
Avellaneda–Stoikov market-making reproduction
The source is Marco Avellaneda and Sasha Stoikov, “High-frequency trading in a limit order book.” The corrected comparison uses the developer-identified 19 July QuantBrain contribution and the developer-attributed Qwen repository. The available task authority is the four-experiment object embedded in the QuantBrain trace.
03
Task authorityCoverage is not directly comparable
QuantBrain attempts the trace-visible analytical, Monte Carlo, robustness, and intensity work. Qwen contains a different four-part outline, and its missing trace prevents prompt-equivalence checks.
Central experimentQwen is numerically closer
Its experiment uses the target arithmetic increments and paper-scale parameters. Fresh low- and medium-γ outcomes are materially closer to the uploaded targets, with small systematic offsets and no predefined acceptance band.
QuantBrain methodThe main simulation is different
It uses geometric Brownian motion, σ=0.02, A=1, and 252 steps instead of arithmetic increments, σ=2, A=140, and 200 steps.
OverallNeither is production-ready
QuantBrain has central theory/calibration errors. Qwen’s reusable strategy and simulator contain severe defects that its green tests never import.
| Article artifact | Bytes / pages | SHA-256 | Material distinction |
Benchmark upload · LimitOrderBook.pdf | 192,083 / 14 | 7a53a062ce5786167b592c457975edf212656615f36d6eae148bf410e63858de | γ={0.01, 0.1, 0.5}; displayed constant spreads 1.33 / 1.29 / 1.15 |
| Later published scan | 433,581 / 8 | 71fa9268339ae82365171caa7416833b3a5d584f05e4f4c087c4ae72d7121a8d | Different γ values, tables, and average-spread convention |
Frozen central experiment: S0=100,
T=1, sigma=2, dt=0.005,
A=140, k=1.5, arithmetic two-point price
increments, 200 decisions, terminal std(q_T), and both
inventory-aware and symmetric strategies. The uploaded bytes and frozen
experiment object are the audit authority.
1Provenance and repository state
Pinned to audited commits
| Property | Developer-attributed Qwen | QuantBrain 397B |
| Pinned target | 111e920a42b9… · single root | 7f58921abdb8… isolated contribution; aba55eda8af2… merged HEAD |
| Older unrelated root | — | 98d4e2de19fea0957e730ad246f11a0b9790bd6c |
| Trace attribution | Supplied run returns 404; developer attribution only | Explicit request says runpod/quantbrain-code-397b-a17b; inherited metadata says runpod/quantbrain |
| Tracked files / size | 33 files · 2.75 MiB in the audited Windows worktree | 80 files / 179.41 MiB isolated; 149 / 181.65 MiB at merged HEAD |
| Supplied tests | 27/27, repeated successfully 5 times | 21/21 isolated; 62/70 at current merged HEAD |
| Dependency state | Lower bounds and unbounded packages despite “pinned” claim | No dependency manifest for the new implementation |
| Result-file integrity | Operational outputs, but documentation/layout drift | Direct narrative/CSV/path contradictions; 98.1% of bytes are six raw-path CSVs |
QuantBrain’s merged HEAD joins unrelated 3 July and 19 July histories with no merge base. Its eight current failures are inherited-test import errors against the new API.
2QuantBrain theory and simulation findings
Source inspection plus independent probes
Two critical analytical errors
- The function called “full spread” subtracts bid distance from ask distance. Quote distances must be added. The implementation simplifies to
2·q·σ²·(T−t), making the result inventory-dependent and sometimes negative.
- The stated first-order condition
λ′(δ)=γλ(δ) is impossible for positive γ and a decreasing exponential intensity. The derivation drops the utility-jump term, then changes formulas without repairing the derivation.
- The supplied spread test encodes the same false inventory-dependent property, so a green test reinforces the error.
The central Monte Carlo changes the task
- Geometric Brownian motion with normal increments replaces the requested arithmetic two-point process.
σ=0.02, A=1, and 252 steps replace σ=2, A=140, and 200 decisions.
1−exp(−λdt) replaces the prescribed λdt Bernoulli approximation.
- Standard deviation of average absolute inventory replaces terminal-inventory dispersion
qT.
| Inventory q | Reported full spread | Ask distance | Bid distance | Interpretation |
| −1 | −4.0 | −2.3333 | 1.6667 | A total bid–ask spread cannot be negative. |
| 0 | 0.0 | −0.3333 | −0.3333 | The target approximation is positive and inventory-independent. |
| 1 | 4.0 | 1.6667 | −2.3333 | The subtraction measures skew, not total spread. |
| Gamma | QuantBrain mean P&L | QuantBrain P&L std | Mean executions | Uploaded-paper inventory mean P&L |
| 0.01 | 0.504 | 1.285 | 0.695 | 66.78 |
| 0.10 | 0.502 | 1.288 | 0.711 | 62.94 |
| 0.50 | 0.491 | 1.337 | 0.791 | 33.92 |
The scale difference is not a tolerance dispute: fewer than one mean execution per path gives the inventory mechanism almost no opportunity to operate.
This is not a claim that every QuantBrain component is wrong. The isolated
contribution has reasonable module boundaries, extensive machine-readable
output, correct long-inventory quote-center direction in its reusable main
simulation helper, a defensible execution-probability transform, deterministic
tests, parseable CSV/JSON, and 20 structurally valid PNGs. The central failure
lies in separate symbolic theory, calibration, price dynamics, and evaluation
metrics.
3Qwen numerical reproduction and its limits
Fresh seed-42 and 20-seed checks
| Gamma / strategy | Uploaded mean / std P&L | Fresh Qwen mean / std P&L | Uploaded / fresh std qT | Assessment |
| 0.01 · inventory | 66.78 / 8.76 | 68.19 / 8.67 | 4.70 / 5.04 | Close at the audited seed. |
| 0.01 · symmetric | 67.36 / 13.40 | 68.17 / 13.13 | 8.65 / 8.58 | Close. |
| 0.10 · inventory | 62.94 / 5.89 | 63.97 / 5.98 | 2.80 / 2.88 | Close. |
| 0.10 · symmetric | 67.21 / 13.43 | 69.07 / 13.01 | 8.66 / 8.77 | Close. |
| 0.50 · inventory | 33.92 / 4.72 | 23.66 / 4.91 | 1.88 / 1.97 | Dispersion is close; mean P&L is about 30% low. |
| 0.50 · symmetric | 66.20 / 14.53 | 67.64 / 13.39 | 9.06 / 9.03 | Broadly close. |
The 20-seed probe shows stability but remains descriptive: no acceptance band was defined before inspection.
Qwen’s deterministic formula module correctly implements the reservation
center and time-dependent total spread, making it sounder than
QuantBrain’s symbolic full_spread function. It does not
reproduce the frozen-inventory/Hamilton–Jacobi–Bellman/utility derivation
or the power-law-intensity comparison visible in QuantBrain’s task object.
Because Qwen’s trace is missing, this is a coverage difference—not a
demonstrated instruction-following failure.
| Gamma | 20-seed inventory mean P&L range | P&L std range | qT std range | What the sensitivity shows |
| 0.01 | 67.9566–69.0514 | 8.5249–9.1826 | 5.0995–5.4593 | Stable, small profit-level offset. |
| 0.10 | 63.8109–64.5711 | 5.8409–6.4439 | 2.7917–3.0603 | Stable and close. |
| 0.50 | 23.1349–23.7446 | 4.6184–4.9480 | 1.8806–2.0480 | The high-γ mean failure is systematic, not a seed-42 accident. |
Main experimentSigned distance is silently replaced
The intensity uses abs(quote−mid). At γ=0.5, q=1, signed λdt=5.9315 becomes 0.08261, heavily suppressing the aggressive liquidation-side fill. Removing abs alone is insufficient: a Bernoulli probability above one needs an explicit policy.
Experiment 2Code and narrative use different scales
The code uses σ=0.02 and produces a 0.00004 spread difference; the report discusses the 0.40 effect that would arise under σ=2.
StatisticsTwo smaller reporting mismatches
The helper uses population rather than the sample standard deviation specified by the QuantBrain-visible frozen plan; Qwen prompt equivalence is unknown. Fresh γ=0.01 inventory std qT=5.04, while the narrative says 5.30.
4Reusable code, tests, documentation, and artifact integrity
Findings outside the headline experiment
Qwen reusable modules are unsafe
strategies.py computes reservation price and ignores it, uses a non-paper half-spread, reverses inventory-skew signs, and can clamp quotes to an arbitrary 0.001 distance.
- A paper-like positive-inventory probe returns quote center 100.4; the required reservation center is 99.6.
simulator.py uses intensity directly as Bernoulli probability without multiplying by dt, fills every advertised path row with the final state, and double-counts inventory value in mark-to-market.
- The 27 tests never import these top-level modules.
QuantBrain reports overstate the checked evidence
RESULTS.md calls σ=0.02 and A=1 paper-matching.
- Its Experiment 3 initial spreads are 1.3708 / 1.3369 / 1.3507; committed CSV values are 3.646471 / 3.960529 / 2.772789.
- It names nonexistent
src/monte_carlo/… paths and uses 21 green tests as methodology validation.
- The isolated tree commits Python bytecode and 184.5 MB of raw paths.
Qwen also documents a nonexistent src/ package and
metrics.py, lists absent CSV outputs, claims a missing MIT
license, and calls lower-bound dependencies “pinned.” Its repository is
operationally healthier, but that does not remove the code defects.
QuantBrain’s structurally valid CSV/JSON files and 20 decodable PNGs show
artifact production, not inferential validity.
5Direct article-one comparison
Qualified artifact-level judgments
| Dimension | Stronger inspected artifact | Why |
| Central numerical reproduction | Qwen-attributed result | Correct target dynamics and calibration with paper-scale output |
| Coverage of the QuantBrain four-part object | QuantBrain 397B | Attempts all four visible topics; task equivalence remains unknown |
| Formula correctness | Qwen, qualified | Sounder formula module, but incomplete analytical coverage |
| Reusable quote/execution layer | QuantBrain, narrowly | Better quote direction and fill probability; wrong experiment calibration |
| Repository and supplied-test health | Qwen | Single history, compact tree, repeatably green tests |
| Test adequacy | Neither | One suite encodes a false property; the other omits broken public modules |
| Report integrity | Qwen, narrowly | Both overclaim; QuantBrain has more direct report/data contradictions |
| Production-ready support | Neither | Each retains material correctness defects |
The Qwen-attributed repository is the better starting point for repairing the central numerical experiment. QuantBrain’s broader architecture is useful, but its “complete reproduction” claim is unsupported by the implemented theory and simulation.
6Paper provenance and automated review
What QCA’s judge could and could not establish
The paper target is version-sensitive
The QCA trace used a 14-page uploaded preprint (SHA-256 7a53a062…8de) with γ={0.01, 0.1, 0.5} and displayed spreads 1.33 / 1.29 / 1.15.
A later eight-page NYU-hosted published scan (SHA-256 71fa9268…a8d) uses γ={0.01, 0.1, 1.0} and average spreads 1.35 / 1.49 / 3.02. The audit evaluates against the uploaded bytes and flags the mismatch as benchmark-design ambiguity.
QuantBrain automated disposition
Partially reproduced
Confidence: medium. This was directionally cautious, but it remained a report-level review.
94,470visible GPT-5.4 tokens
39.6sjudge duration
22 / 21total reads / unique result files
0source files or tests executed
Seven reads hit the 8,000-character ceiling. No equivalent Qwen judgment is auditable because its supplied run is absent from the public share.
What the judge noticed
- Weak empirical risk-control improvements and sparse executions.
- Experiment 3 parameter mismatch and use of average path inventory.
- No formal inference and overstated summary language.
- A result too mixed for “fully reproduced.”
What source inspection added
Wrong full-spread operation
Impossible first-order condition
Wrong stochastic process
Merged HEAD test failures
Unrelated-history merge
Report/CSV contradictions
Nonexistent source paths
Missing dependency manifest
Trace detail: QCA requested runpod/quantbrain-code-397b-a17b
for job job-qca-runner-gigifko; the status field says
Succeeded. The root took 856.594 seconds, including 815.332
seconds waiting for the coding job. Inherited metadata instead says
runpod/quantbrain and reuses the 17 July thread ID. The generic
four-level judge rubric has no numeric score and no explicit tolerances for
this study’s P&L dispersion, terminal-inventory dispersion, spread, or
fill-rate quantities. Recorded cost is null, and the 94,470
tokens exclude tested-model generation.
| Trace | Job outcome | Scope | Visible verdict | Visible tokens | Why it is not a common baseline |
| 17 Jul · QuantBrain 397B | Failed | 4 experiments | Partial / High | 70,095 | Pipeline still judged files already on the mutable default branch. |
| 17 Jul · generic QuantBrain | Succeeded | 3 experiments | Partial / Medium | 52,020 | Different model string and task scope. |
| 13 Jul · Sonnet 4.6 | Succeeded | 2 experiments | Partial / Medium in prose | 40,084 | Different scope and smaller evidence read. |
| 19 Jul · QuantBrain 397B | Succeeded | 4 experiments | Partial / Medium | 94,470 | Corrected primary QuantBrain run; tested-model exchange remains opaque. |
The failed 17 July job received the same PARTIALLY REPRODUCED verdict with higher stated confidence than the later successful job. Job success and generated-commit identity must be hard evaluation gates.
Study 02 · complete findings
skfolio portfolio-optimization reproduction
The source is Carlo Nicolini, Matteo Manzi, and Hugo Delatte, “skfolio: Portfolio Optimization in Python.” It defines three workflows: a constrained minimum-variance frontier, a five-factor prior, and a stacked portfolio evaluated through walk-forward splits.
04
| Captured article version | Bytes / pages | SHA-256 | Role in the audit |
| skfolio paper · arXiv v1 | 794,465 / 7 | d9ab3aea73486cdbe606e28cdb3ed49a790fa8e6b873ee3d37a518794e5cc1bf | Earlier captured version |
| skfolio paper · arXiv v2 | 800,224 / 7 | b04c86f6ff4b57406480271dee6705a8dc742af4bf6b9e2095692e5d9ae2fd8f | Current well before the run and highly likely, but not byte-pinned by QCA |
The article presents three example workflows rather than invariant target
returns: a 100-point minimum-variance frontier with an 80/20 chronological
split, AAPL capped at 20%, and L2 coefficient 0.01; a five-factor
MeanRisk prior with Ridge(alpha=0.1); and a
252/60 held-out walk-forward comparison that stacks inverse volatility,
MeanRisk/CVaR, and default hierarchical risk parity against equal weight.
It does not require stacking to outperform equal weight or define fixed
metrics for future data and dependency versions.
Qwen evidenceExperiment 3 is recoverable
Held-out prediction tables, loadable population objects, and JSON metrics/temporal validation are committed and internally reproducible.
Qwen defectExperiments 1 and 2 duplicate train
The evaluator accepts train and test data but ignores both. It measures the same train-backed population twice and labels one copy “test.”
QuantBrain structureExperiment 1 predicts separately
This structure is sounder, but 21/100 nearest-risk comparisons and train_dominates_test=false are narrated as confirmation.
QuantBrain workflowThe all-run path aborts
A wrong function signature raises TypeError before Experiment 3. Named result files are absent from the repository.
1Repository construction and evidence inventory
Developer-attributed artifacts
| Property | Qwen-attributed artifact | QuantBrain-attributed artifact |
| Pinned commit | fd217ac9e205f0102d2a29c5d189daaf45806bab | 274681ced761a5d952ac7a1c5e9520bf81a7c569 |
| Tracked files / bytes | 39 · 1,053,483 bytes | 14 · 109,024 bytes |
| Committed result evidence | 23 artifacts plus RESULTS.md | .gitkeep plus RESULTS.md |
| Supplied tests | 124/124 under current resolution | 27/28 current; 28/28 compatible, with exception-message drift |
| Workflow exercised by tests | Experiment functions covered; no semantic oracle for use of X_test | Experiment entry points and all-run command are not called |
| Dependency record | Partial package/version JSON; no exact lock | Narrative claim only; no exact lock |
| Public trace | Requested Qwen 397B and job status reported Succeeded | Supplied run returns HTTP 404 in supplied share |
2Experiment 1 · constrained efficient frontier
100 portfolios, chronological 80/20 split
Qwen · decisive defectDataset arguments are unused
The committed train and “test” files are byte-identical. Correct test prediction changes both expected return and variance.
QuantBrain · contradictionThe reported conclusion is reversed
The source records train_dominates_test=false after 21/100 nearest-risk comparisons, yet the static report calls the experiment successful.
Independent reconstructionThe current-data comparison is negative
Risk interpolation gives 73 comparable points, 16 with train at least as high, and mean annualized train-minus-test gap −0.000571.
| Frontier-wide mean | Qwen committed train | Qwen committed “test” | Qwen corrected test | QuantBrain fresh train / test |
| Daily expected return | 0.000758524 | 0.000758524 | 0.000781840 | 0.000758525 / 0.000781842 |
| Daily variance | 0.000142454 | 0.000142454 | 0.000146872 | Separately evaluated |
| AAPL cap violations | 0 | 0 | 0 | 0 |
Additional Qwen method mismatch
The saved red and blue train/test plot series lie exactly on top of one another, while the narrative claims a visible quantitative gap. The plot also uses daily expected return on the x-axis and daily variance on the y-axis; the paper uses annualized standard deviation on x and annualized mean return on y. The optimizer construction is plausible, but the visualization and OOS claim are not faithful.
QuantBrain report versus fresh execution
The static report claims train annualized mean 18.20–25.89% and test 10.86–12.95%. Fresh committed source produces 18.20–20.03% and 19.10–20.30%. Reported risk ranges are likewise stale or inconsistent.
QuantBrain’s negative train/test result is not necessarily an
implementation failure: dataset and package versions can change frontier
geometry. The supported report failure is narrower and direct—the source
Boolean and fresh comparison are negative while the narrative calls the
experiment confirmed, without frozen bytes or solver versions to explain
the drift.
3Experiment 2 · factor-model prior
Five factors, Ridge α=0.1
Qwen: fitted prior, wrong evidence path
- The optimizer has a fitted
prior_estimator_, and a 20×5 loading matrix is saved.
prior_attached=false is a validator false negative because it checks the output population rather than the optimizer.
- The train and “test” tables are byte-identical: daily mean return 0.000685272 and variance 0.000134033.
- Correct test prediction gives return 0.000650865 and variance 0.000118411.
fit_intercept=False differs from the paper’s displayed default; measured weight impact is small (L1 0.000319, maximum component 0.0000595).
QuantBrain: fitted model, broken diagnostics
- The factor prior fits and a 20×5 loading matrix exists in memory.
predict(X_test, y_test) uses an invalid signature.
- The loading matrix is sought on the wrong attribute.
- Missing
solver_status is interpreted as failed integration.
- No out-of-sample metrics are produced; JSON Booleans are serialized as strings.
4Experiment 3 · stacked portfolio with walk-forward evaluation
Strongest Qwen evidence; clearest QuantBrain scope failure
| Saved metric | Qwen stacking | Qwen equal weight | Observed trade-off |
| Daily mean return | 0.000471409 | 0.000771023 | Equal weight higher. |
| Annualized volatility | 0.179593 | 0.204180 | Stacking lower. |
| Sharpe ratio | 0.661469 | 0.951599 | Equal weight higher. |
| CVaR · positive loss magnitude | 0.027435 | 0.030968 | Stacking lower. |
| Max drawdown · skfolio convention | 0.309954 | 0.346955 | Stacking lower. |
| Simple geometric drawdown · independently recomputed | 0.287423 | 0.316756 | Stacking lower under this second convention. |
Saved total_return | 0.650545 | 1.064012 | Actually a non-compounded cumulative sum. |
| Independently compounded gain | 0.754223 | 1.584426 | 75.42% versus 158.44% gain. |
The paper does not require stacking to outperform equal weight. Lower return and Sharpe are outcomes, not reproduction failures; the relevant question is whether the method and held-out evaluation were reproduced.
Qwen: genuine outer evaluation, qualified
- The outer 252/60 walk-forward runs on the 1,663-row held-out set.
- It creates 23 complete periods and 1,380 out-of-sample daily returns on matching dates for both portfolio strategies.
- The covered dates are 2017-05-23 through 2022-11-11; the final 31 held-out observations do not form another complete 60-row test window and are unused.
- Predictions and fitted populations are saved and reload successfully.
- HRP uses CVaR rather than the paper’s default risk measure.
- Internal
cv=None produces default five-fold K-fold, matching the paper’s omitted/default setting but remaining a non-temporal caveat.
QuantBrain: wrong outer scope and blocked runner
- The code evaluates all 8,312 rows rather than only the 1,663-row held-out set, yielding 134 rather than 23 folds.
- Internal CV is explicitly changed to three-fold K-fold.
- A fold validator reports a false negative because it condemns intentional overlap between adjacent rolling training windows; all 134 actual folds still satisfy
train_end < test_start.
- The all-run command passes
results_dir to a no-argument function and aborts with TypeError before Experiment 3.
- Even after that signature fix, report-schema keys do not match the runner output.
5Tests, dependencies, and automated review
Green tests do not answer the central semantic questions
Fresh test results and coverage gaps
- Qwen: 34 + 51 + 39 = 124 current passes. The audit did not directly execute its top-level runner.
- QuantBrain: 27/28 under the current unpinned environment; 28/28 under a compatible skfolio 0.6 environment. The current failure is exception-message drift.
- Fifteen compatible Qwen Experiment 3 cases repeat expensive walk-forward work, including six full end-to-end runs; the suite is broad but inefficient.
- Qwen tests never prove that test metrics are derived from
X_test.
- QuantBrain tests never call experiment entry points or the all-run command.
- Neither repository has a complete lock. Qwen’s committed environment omits CVXPY/solver details; the current resolution is skfolio 0.20.1 with CVXPY 1.9.2, while the compatible reconstruction uses skfolio 0.6.0 with CVXPY 1.6.7. Combining skfolio 0.6.0 with the newest resolved CVXPY 1.9.2 fails in this reconstruction. Bundled dataset bytes are also unpinned.
Qwen automated disposition
Partially reproduced
Confidence: medium. The accessible judge correctly saw duplicate tables and a contradictory prior flag, but did not inspect source or execute code.
Visible root usage comprises 39,074 prompt and 7,137 completion tokens. It excludes the external Qwen repository job and PDF analysis, and the trace records no complete cost.
46,211visible QCA/GPT-5.4 tokens
26.8sfinal-review duration
12result/report files read
0source files or tests executed
What the accessible trace establishes
- Requested model:
runpod/qwen3.5-397b-a17b.
- External job
job-qca-runner-p8p92jz reported Succeeded.
- Full root workflow: 61m 59.668s; external job: 58m 08.411s.
- The judge was reasonably blind to model identity in its rendered message.
- Visible usage includes 10,240 cache-read tokens but excludes Qwen repository generation and PDF-tool usage; complete cost is unrecoverable.
What the judge did not inspect
Source code
Tests and runner
Environment lock
Solver logs
Saved optimizer objects
Fold schedule
Predictions and weights
Plots through visual review
6Direct article-two comparison
Qualified artifact-level judgments
| Dimension | Stronger inspected artifact | Why |
| Recoverable evidence | Qwen-attributed result | Persisted predictions, objects, tables, and plots |
| Experiment 1 evaluation structure | QuantBrain-attributed result | Separate train/test predictions rather than duplicate outputs |
| Experiment 1 conclusion integrity | Neither | Qwen does not test out of sample; QuantBrain reverses its actual negative result |
| Experiment 2 evidence | Qwen-attributed result | Prior and loadings are recoverable despite a broken validation flag |
| Experiment 3 article fidelity | Qwen-attributed result | Correct outer held-out scope and 23 walk-forward periods |
| End-to-end runner | Qwen, qualified | QuantBrain’s runner aborts; Qwen’s expensive root workflow was not independently rerun |
| Complete reproduction | Neither | Both retain method, verification, and environment gaps |
The Qwen artifact is the more recoverable basis for continued work. QuantBrain’s separate train/test structure in Experiment 1 is a real strength, but its report, runner, and missing evidence require substantial repair.
The supplied QuantBrain run ID returns HTTP 404 under the supplied share
token. Its exact task, requested model, job status, judge evidence,
verdict, and usage cannot be verified. The paper trace retained an
unversioned arXiv URL; version 2 was current long before the run and is
highly likely, but the fetched PDF hash was not recorded by QCA. The
generated task also did not freeze dataset bytes, solver, dependency
versions, or an acceptance rule. Its generated narrative mixes the 2,263-row
factor dataset with the 8,312-row S&P 500 workflows, another sign that
summary text is not a reliable substitute for same-run artifacts.