Independent repository investigation · 19–20 July 2026

QCA article reproductions: results and alternatives

A standalone, source-pinned review of two QCA article-to-repository studies: the QuantBrain-attributed result and Qwen-attributed alternative for each article, including executable behavior, numerical evidence, paper fidelity, test coverage, and provenance.

Executive conclusion

Both workflows produced useful artifacts; neither comparison supports a causal model ranking.

In the Avellaneda–Stoikov study, the developer-attributed Qwen repository is materially closer on the central Monte Carlo experiment, while QuantBrain 397B attempts more of the four-part analytical scope. Both contain serious untested correctness defects.

In the skfolio study, the Qwen-attributed repository preserves more recoverable evidence and a genuine held-out walk-forward result. QuantBrain has the sounder train/test structure in Experiment 1, but its report reverses the observed result and its all-run command fails before Experiment 3. Neither study is sufficiently controlled to infer general model capability.

Study 01 · market making

Avellaneda–Stoikov reproduction

A four-part theory and Monte Carlo exercise built around an uploaded preprint whose numerical targets differ from a later published scan.

Best-supported result
Qwen central Monte Carlo: seed 42 plus 20-seed sensitivity
Broader attempted scope
QuantBrain 397B
Clean model ranking
Not supported
Study 02 · portfolio optimization

skfolio reproduction

Three workflows covering a constrained efficient frontier, a factor-model prior, and stacked portfolio optimization with walk-forward evaluation.

Most recoverable artifact
Qwen-attributed repository
Sounder Exp. 1 structure
QuantBrain-attributed repository
Complete reproduction
Neither artifact
Article reproductions2

Market making and portfolio optimization.

Repository artifacts4

Two developer-attributed artifacts per QCA research study.

Missing comparator traces2

One supplied comparison run is unavailable in each public share.

Universal rankings supported0

Tasks, prompts, and provenance are not controlled tightly enough.

Terminology. The retained artifacts identify the system as QCA, not a separate “QSA” system. “Result” means the QuantBrain-attributed repository and “alternative” means the Qwen-attributed comparator, regardless of which link arrived first.
Responsibility boundary

QCA orchestrated the workflow; the coding models generated repositories.

The developer stated on 17 July that the language models were used only for repository generation and that the remaining behavior came from QCA presets. The retained traces support that division, with important attribution limits.

01
QCA

Task and experiment design

QCA acquired the article, produced the experiment object, selected presets, and dispatched the repository job. The skfolio PDF-analysis stage specifically requested GPT-5.4.

Coding model

Repository implementation

The developer attributes the repository contents to the requested QuantBrain or Qwen coding model. Git commits use the neutral author QCA Agent.

QCA judge

Automated assessment

GPT-5.4 reviewed selected generated files and returned a reproduction disposition. Those reviews did not substitute for source execution or independent scientific verification.

Workflow stageEvidence-based attribution
Article acquisition and parsingQCA workflow; the skfolio PDF-analysis tool specifically requested GPT-5.4
Experiment-object generationQCA workflow / GPT-5.4 stage
Job dispatch and terminal-state handlingQCA orchestration
Repository implementationDeveloper-attributed coding model
Automated repository judgmentQCA workflow / GPT-5.4 judge
Independent checks on this pageThis repository audit
The traces sometimes preserve a requested model identifier and terminal job status, but not the tested model’s complete exchange, tool loop, retries, token usage, price, or server-returned identity. Requested IDs and developer labels are not checkpoint-level attestation.
Investigation map

Two source articles, one result and one alternative for each.

“Result” and “alternative” denote the developer-attributed QuantBrain and Qwen artifacts, not a randomized treatment/control design. The Qwen skfolio links happened to arrive first.

02
QCA reproductionSource articleQuantBrain-attributed resultQwen-attributed alternativeTrace situationArtifact-level outcome
1Avellaneda–Stoikov market makingQuantBrain 397B repositoryQwen-attributed repositoryCorrected QuantBrain trace accessible; Qwen run absent from shareQwen has the meaningful central numerical reproduction; QuantBrain attempts broader scope
2skfolio portfolio optimizationQuantBrain-attributed repositoryQwen-attributed repositoryQwen trace accessible; QuantBrain run absent from shareQwen is more recoverable overall; QuantBrain has one better evaluation structure but a broken evidence path
A

Artifact comparison matrix

Different questions require different evidence

DimensionAvellaneda–Stoikov · QwenAvellaneda–Stoikov · QuantBrain 397Bskfolio · Qwenskfolio · QuantBrain
Central executable result Paper-scale Monte Carlo; low/medium γ materially closer to uploaded targets, high γ mean fails Different process and calibration; about 0.5 mean P&L Real walk-forward result; Exp. 1/2 OOS tables invalid Runner aborts before Exp. 3; claimed outputs absent
Scope coverage Different, narrower four-part outline; task equivalence is unknown Broad attempt at all four topics visible in its trace All three generated workflows are represented All three modules exist, although the runner cannot complete them
Method fidelity Target process and sounder formula check; signed-distance and reusable-API defects remain Central symbolic, process, calibration, and metric mismatches Paper-aligned outer Exp. 3; one fidelity deviation plus a non-temporal default-CV caveat Wrong Exp. 3 outer scope and changed internal CV
Repository evidence Single root, 27/27 tests, parseable outputs; tests miss reusable API Unrelated histories merged; HEAD 62/70; report/data conflicts Committed predictions, objects, tables, plots; 124 tests Narrative plus .gitkeep; no claimed result files
Public trace coverage Supplied run absent from share; attribution is developer-provided Requested model and successful status visible; model exchange absent Requested model and reported success visible; generation opaque Supplied run absent from share
Artifact-level assessment More useful numerical starting point, with major repairs required Broader architecture, but claims are not supported by current files More recoverable starting point, not a complete reproduction Useful structural ideas, but workflow and evidence require repair

“Qwen” is shorthand for developer attribution where its public trace is unavailable. skfolio’s QuantBrain model ID is likewise not verifiable from the supplied public share.

Study 01 · complete findings

Avellaneda–Stoikov market-making reproduction

The source is Marco Avellaneda and Sasha Stoikov, “High-frequency trading in a limit order book.” The corrected comparison uses the developer-identified 19 July QuantBrain contribution and the developer-attributed Qwen repository. The available task authority is the four-experiment object embedded in the QuantBrain trace.

03
Task authority

Coverage is not directly comparable

QuantBrain attempts the trace-visible analytical, Monte Carlo, robustness, and intensity work. Qwen contains a different four-part outline, and its missing trace prevents prompt-equivalence checks.

Central experiment

Qwen is numerically closer

Its experiment uses the target arithmetic increments and paper-scale parameters. Fresh low- and medium-γ outcomes are materially closer to the uploaded targets, with small systematic offsets and no predefined acceptance band.

QuantBrain method

The main simulation is different

It uses geometric Brownian motion, σ=0.02, A=1, and 252 steps instead of arithmetic increments, σ=2, A=140, and 200 steps.

Overall

Neither is production-ready

QuantBrain has central theory/calibration errors. Qwen’s reusable strategy and simulator contain severe defects that its green tests never import.

Article artifactBytes / pagesSHA-256Material distinction
Benchmark upload · LimitOrderBook.pdf192,083 / 147a53a062ce5786167b592c457975edf212656615f36d6eae148bf410e63858deγ={0.01, 0.1, 0.5}; displayed constant spreads 1.33 / 1.29 / 1.15
Later published scan433,581 / 871fa9268339ae82365171caa7416833b3a5d584f05e4f4c087c4ae72d7121a8dDifferent γ values, tables, and average-spread convention
Frozen central experiment: S0=100, T=1, sigma=2, dt=0.005, A=140, k=1.5, arithmetic two-point price increments, 200 decisions, terminal std(q_T), and both inventory-aware and symmetric strategies. The uploaded bytes and frozen experiment object are the audit authority.
1

Provenance and repository state

Pinned to audited commits

PropertyDeveloper-attributed QwenQuantBrain 397B
Pinned target111e920a42b9… · single root7f58921abdb8… isolated contribution; aba55eda8af2… merged HEAD
Older unrelated root98d4e2de19fea0957e730ad246f11a0b9790bd6c
Trace attributionSupplied run returns 404; developer attribution onlyExplicit request says runpod/quantbrain-code-397b-a17b; inherited metadata says runpod/quantbrain
Tracked files / size33 files · 2.75 MiB in the audited Windows worktree80 files / 179.41 MiB isolated; 149 / 181.65 MiB at merged HEAD
Supplied tests27/27, repeated successfully 5 times21/21 isolated; 62/70 at current merged HEAD
Dependency stateLower bounds and unbounded packages despite “pinned” claimNo dependency manifest for the new implementation
Result-file integrityOperational outputs, but documentation/layout driftDirect narrative/CSV/path contradictions; 98.1% of bytes are six raw-path CSVs

QuantBrain’s merged HEAD joins unrelated 3 July and 19 July histories with no merge base. Its eight current failures are inherited-test import errors against the new API.

2

QuantBrain theory and simulation findings

Source inspection plus independent probes

Two critical analytical errors

  • The function called “full spread” subtracts bid distance from ask distance. Quote distances must be added. The implementation simplifies to 2·q·σ²·(T−t), making the result inventory-dependent and sometimes negative.
  • The stated first-order condition λ′(δ)=γλ(δ) is impossible for positive γ and a decreasing exponential intensity. The derivation drops the utility-jump term, then changes formulas without repairing the derivation.
  • The supplied spread test encodes the same false inventory-dependent property, so a green test reinforces the error.

The central Monte Carlo changes the task

  • Geometric Brownian motion with normal increments replaces the requested arithmetic two-point process.
  • σ=0.02, A=1, and 252 steps replace σ=2, A=140, and 200 decisions.
  • 1−exp(−λdt) replaces the prescribed λdt Bernoulli approximation.
  • Standard deviation of average absolute inventory replaces terminal-inventory dispersion qT.
Inventory qReported full spreadAsk distanceBid distanceInterpretation
−1−4.0−2.33331.6667A total bid–ask spread cannot be negative.
00.0−0.3333−0.3333The target approximation is positive and inventory-independent.
14.01.6667−2.3333The subtraction measures skew, not total spread.
GammaQuantBrain mean P&LQuantBrain P&L stdMean executionsUploaded-paper inventory mean P&L
0.010.5041.2850.69566.78
0.100.5021.2880.71162.94
0.500.4911.3370.79133.92

The scale difference is not a tolerance dispute: fewer than one mean execution per path gives the inventory mechanism almost no opportunity to operate.

This is not a claim that every QuantBrain component is wrong. The isolated contribution has reasonable module boundaries, extensive machine-readable output, correct long-inventory quote-center direction in its reusable main simulation helper, a defensible execution-probability transform, deterministic tests, parseable CSV/JSON, and 20 structurally valid PNGs. The central failure lies in separate symbolic theory, calibration, price dynamics, and evaluation metrics.
3

Qwen numerical reproduction and its limits

Fresh seed-42 and 20-seed checks

Gamma / strategyUploaded mean / std P&LFresh Qwen mean / std P&LUploaded / fresh std qTAssessment
0.01 · inventory66.78 / 8.7668.19 / 8.674.70 / 5.04Close at the audited seed.
0.01 · symmetric67.36 / 13.4068.17 / 13.138.65 / 8.58Close.
0.10 · inventory62.94 / 5.8963.97 / 5.982.80 / 2.88Close.
0.10 · symmetric67.21 / 13.4369.07 / 13.018.66 / 8.77Close.
0.50 · inventory33.92 / 4.7223.66 / 4.911.88 / 1.97Dispersion is close; mean P&L is about 30% low.
0.50 · symmetric66.20 / 14.5367.64 / 13.399.06 / 9.03Broadly close.

The 20-seed probe shows stability but remains descriptive: no acceptance band was defined before inspection.

Qwen’s deterministic formula module correctly implements the reservation center and time-dependent total spread, making it sounder than QuantBrain’s symbolic full_spread function. It does not reproduce the frozen-inventory/Hamilton–Jacobi–Bellman/utility derivation or the power-law-intensity comparison visible in QuantBrain’s task object. Because Qwen’s trace is missing, this is a coverage difference—not a demonstrated instruction-following failure.
Gamma20-seed inventory mean P&L rangeP&L std rangeqT std rangeWhat the sensitivity shows
0.0167.9566–69.05148.5249–9.18265.0995–5.4593Stable, small profit-level offset.
0.1063.8109–64.57115.8409–6.44392.7917–3.0603Stable and close.
0.5023.1349–23.74464.6184–4.94801.8806–2.0480The high-γ mean failure is systematic, not a seed-42 accident.
Main experiment

Signed distance is silently replaced

The intensity uses abs(quote−mid). At γ=0.5, q=1, signed λdt=5.9315 becomes 0.08261, heavily suppressing the aggressive liquidation-side fill. Removing abs alone is insufficient: a Bernoulli probability above one needs an explicit policy.

Experiment 2

Code and narrative use different scales

The code uses σ=0.02 and produces a 0.00004 spread difference; the report discusses the 0.40 effect that would arise under σ=2.

Statistics

Two smaller reporting mismatches

The helper uses population rather than the sample standard deviation specified by the QuantBrain-visible frozen plan; Qwen prompt equivalence is unknown. Fresh γ=0.01 inventory std qT=5.04, while the narrative says 5.30.

4

Reusable code, tests, documentation, and artifact integrity

Findings outside the headline experiment

Qwen reusable modules are unsafe

  • strategies.py computes reservation price and ignores it, uses a non-paper half-spread, reverses inventory-skew signs, and can clamp quotes to an arbitrary 0.001 distance.
  • A paper-like positive-inventory probe returns quote center 100.4; the required reservation center is 99.6.
  • simulator.py uses intensity directly as Bernoulli probability without multiplying by dt, fills every advertised path row with the final state, and double-counts inventory value in mark-to-market.
  • The 27 tests never import these top-level modules.

QuantBrain reports overstate the checked evidence

  • RESULTS.md calls σ=0.02 and A=1 paper-matching.
  • Its Experiment 3 initial spreads are 1.3708 / 1.3369 / 1.3507; committed CSV values are 3.646471 / 3.960529 / 2.772789.
  • It names nonexistent src/monte_carlo/… paths and uses 21 green tests as methodology validation.
  • The isolated tree commits Python bytecode and 184.5 MB of raw paths.
Qwen also documents a nonexistent src/ package and metrics.py, lists absent CSV outputs, claims a missing MIT license, and calls lower-bound dependencies “pinned.” Its repository is operationally healthier, but that does not remove the code defects. QuantBrain’s structurally valid CSV/JSON files and 20 decodable PNGs show artifact production, not inferential validity.
5

Direct article-one comparison

Qualified artifact-level judgments

DimensionStronger inspected artifactWhy
Central numerical reproductionQwen-attributed resultCorrect target dynamics and calibration with paper-scale output
Coverage of the QuantBrain four-part objectQuantBrain 397BAttempts all four visible topics; task equivalence remains unknown
Formula correctnessQwen, qualifiedSounder formula module, but incomplete analytical coverage
Reusable quote/execution layerQuantBrain, narrowlyBetter quote direction and fill probability; wrong experiment calibration
Repository and supplied-test healthQwenSingle history, compact tree, repeatably green tests
Test adequacyNeitherOne suite encodes a false property; the other omits broken public modules
Report integrityQwen, narrowlyBoth overclaim; QuantBrain has more direct report/data contradictions
Production-ready supportNeitherEach retains material correctness defects

The Qwen-attributed repository is the better starting point for repairing the central numerical experiment. QuantBrain’s broader architecture is useful, but its “complete reproduction” claim is unsupported by the implemented theory and simulation.

6

Paper provenance and automated review

What QCA’s judge could and could not establish

The paper target is version-sensitive

The QCA trace used a 14-page uploaded preprint (SHA-256 7a53a062…8de) with γ={0.01, 0.1, 0.5} and displayed spreads 1.33 / 1.29 / 1.15.

A later eight-page NYU-hosted published scan (SHA-256 71fa9268…a8d) uses γ={0.01, 0.1, 1.0} and average spreads 1.35 / 1.49 / 3.02. The audit evaluates against the uploaded bytes and flags the mismatch as benchmark-design ambiguity.

QuantBrain automated disposition

Partially reproduced

Confidence: medium. This was directionally cautious, but it remained a report-level review.

94,470visible GPT-5.4 tokens
39.6sjudge duration
22 / 21total reads / unique result files
0source files or tests executed

Seven reads hit the 8,000-character ceiling. No equivalent Qwen judgment is auditable because its supplied run is absent from the public share.

What the judge noticed

  • Weak empirical risk-control improvements and sparse executions.
  • Experiment 3 parameter mismatch and use of average path inventory.
  • No formal inference and overstated summary language.
  • A result too mixed for “fully reproduced.”

What source inspection added

Wrong full-spread operation
Impossible first-order condition
Wrong stochastic process
Merged HEAD test failures
Unrelated-history merge
Report/CSV contradictions
Nonexistent source paths
Missing dependency manifest
Trace detail: QCA requested runpod/quantbrain-code-397b-a17b for job job-qca-runner-gigifko; the status field says Succeeded. The root took 856.594 seconds, including 815.332 seconds waiting for the coding job. Inherited metadata instead says runpod/quantbrain and reuses the 17 July thread ID. The generic four-level judge rubric has no numeric score and no explicit tolerances for this study’s P&L dispersion, terminal-inventory dispersion, spread, or fill-rate quantities. Recorded cost is null, and the 94,470 tokens exclude tested-model generation.
TraceJob outcomeScopeVisible verdictVisible tokensWhy it is not a common baseline
17 Jul · QuantBrain 397BFailed4 experimentsPartial / High70,095Pipeline still judged files already on the mutable default branch.
17 Jul · generic QuantBrainSucceeded3 experimentsPartial / Medium52,020Different model string and task scope.
13 Jul · Sonnet 4.6Succeeded2 experimentsPartial / Medium in prose40,084Different scope and smaller evidence read.
19 Jul · QuantBrain 397BSucceeded4 experimentsPartial / Medium94,470Corrected primary QuantBrain run; tested-model exchange remains opaque.

The failed 17 July job received the same PARTIALLY REPRODUCED verdict with higher stated confidence than the later successful job. Job success and generated-commit identity must be hard evaluation gates.

Study 02 · complete findings

skfolio portfolio-optimization reproduction

The source is Carlo Nicolini, Matteo Manzi, and Hugo Delatte, “skfolio: Portfolio Optimization in Python.” It defines three workflows: a constrained minimum-variance frontier, a five-factor prior, and a stacked portfolio evaluated through walk-forward splits.

04
Captured article versionBytes / pagesSHA-256Role in the audit
skfolio paper · arXiv v1794,465 / 7d9ab3aea73486cdbe606e28cdb3ed49a790fa8e6b873ee3d37a518794e5cc1bfEarlier captured version
skfolio paper · arXiv v2800,224 / 7b04c86f6ff4b57406480271dee6705a8dc742af4bf6b9e2095692e5d9ae2fd8fCurrent well before the run and highly likely, but not byte-pinned by QCA
The article presents three example workflows rather than invariant target returns: a 100-point minimum-variance frontier with an 80/20 chronological split, AAPL capped at 20%, and L2 coefficient 0.01; a five-factor MeanRisk prior with Ridge(alpha=0.1); and a 252/60 held-out walk-forward comparison that stacks inverse volatility, MeanRisk/CVaR, and default hierarchical risk parity against equal weight. It does not require stacking to outperform equal weight or define fixed metrics for future data and dependency versions.
Qwen evidence

Experiment 3 is recoverable

Held-out prediction tables, loadable population objects, and JSON metrics/temporal validation are committed and internally reproducible.

Qwen defect

Experiments 1 and 2 duplicate train

The evaluator accepts train and test data but ignores both. It measures the same train-backed population twice and labels one copy “test.”

QuantBrain structure

Experiment 1 predicts separately

This structure is sounder, but 21/100 nearest-risk comparisons and train_dominates_test=false are narrated as confirmation.

QuantBrain workflow

The all-run path aborts

A wrong function signature raises TypeError before Experiment 3. Named result files are absent from the repository.

1

Repository construction and evidence inventory

Developer-attributed artifacts

PropertyQwen-attributed artifactQuantBrain-attributed artifact
Pinned commitfd217ac9e205f0102d2a29c5d189daaf45806bab274681ced761a5d952ac7a1c5e9520bf81a7c569
Tracked files / bytes39 · 1,053,483 bytes14 · 109,024 bytes
Committed result evidence23 artifacts plus RESULTS.md.gitkeep plus RESULTS.md
Supplied tests124/124 under current resolution27/28 current; 28/28 compatible, with exception-message drift
Workflow exercised by testsExperiment functions covered; no semantic oracle for use of X_testExperiment entry points and all-run command are not called
Dependency recordPartial package/version JSON; no exact lockNarrative claim only; no exact lock
Public traceRequested Qwen 397B and job status reported SucceededSupplied run returns HTTP 404 in supplied share
2

Experiment 1 · constrained efficient frontier

100 portfolios, chronological 80/20 split

Qwen · decisive defect

Dataset arguments are unused

The committed train and “test” files are byte-identical. Correct test prediction changes both expected return and variance.

QuantBrain · contradiction

The reported conclusion is reversed

The source records train_dominates_test=false after 21/100 nearest-risk comparisons, yet the static report calls the experiment successful.

Independent reconstruction

The current-data comparison is negative

Risk interpolation gives 73 comparable points, 16 with train at least as high, and mean annualized train-minus-test gap −0.000571.

Frontier-wide meanQwen committed trainQwen committed “test”Qwen corrected testQuantBrain fresh train / test
Daily expected return0.0007585240.0007585240.0007818400.000758525 / 0.000781842
Daily variance0.0001424540.0001424540.000146872Separately evaluated
AAPL cap violations0000

Additional Qwen method mismatch

The saved red and blue train/test plot series lie exactly on top of one another, while the narrative claims a visible quantitative gap. The plot also uses daily expected return on the x-axis and daily variance on the y-axis; the paper uses annualized standard deviation on x and annualized mean return on y. The optimizer construction is plausible, but the visualization and OOS claim are not faithful.

QuantBrain report versus fresh execution

The static report claims train annualized mean 18.20–25.89% and test 10.86–12.95%. Fresh committed source produces 18.20–20.03% and 19.10–20.30%. Reported risk ranges are likewise stale or inconsistent.

QuantBrain’s negative train/test result is not necessarily an implementation failure: dataset and package versions can change frontier geometry. The supported report failure is narrower and direct—the source Boolean and fresh comparison are negative while the narrative calls the experiment confirmed, without frozen bytes or solver versions to explain the drift.
3

Experiment 2 · factor-model prior

Five factors, Ridge α=0.1

Qwen: fitted prior, wrong evidence path

  • The optimizer has a fitted prior_estimator_, and a 20×5 loading matrix is saved.
  • prior_attached=false is a validator false negative because it checks the output population rather than the optimizer.
  • The train and “test” tables are byte-identical: daily mean return 0.000685272 and variance 0.000134033.
  • Correct test prediction gives return 0.000650865 and variance 0.000118411.
  • fit_intercept=False differs from the paper’s displayed default; measured weight impact is small (L1 0.000319, maximum component 0.0000595).

QuantBrain: fitted model, broken diagnostics

  • The factor prior fits and a 20×5 loading matrix exists in memory.
  • predict(X_test, y_test) uses an invalid signature.
  • The loading matrix is sought on the wrong attribute.
  • Missing solver_status is interpreted as failed integration.
  • No out-of-sample metrics are produced; JSON Booleans are serialized as strings.
4

Experiment 3 · stacked portfolio with walk-forward evaluation

Strongest Qwen evidence; clearest QuantBrain scope failure

Saved metricQwen stackingQwen equal weightObserved trade-off
Daily mean return0.0004714090.000771023Equal weight higher.
Annualized volatility0.1795930.204180Stacking lower.
Sharpe ratio0.6614690.951599Equal weight higher.
CVaR · positive loss magnitude0.0274350.030968Stacking lower.
Max drawdown · skfolio convention0.3099540.346955Stacking lower.
Simple geometric drawdown · independently recomputed0.2874230.316756Stacking lower under this second convention.
Saved total_return0.6505451.064012Actually a non-compounded cumulative sum.
Independently compounded gain0.7542231.58442675.42% versus 158.44% gain.

The paper does not require stacking to outperform equal weight. Lower return and Sharpe are outcomes, not reproduction failures; the relevant question is whether the method and held-out evaluation were reproduced.

Qwen: genuine outer evaluation, qualified

  • The outer 252/60 walk-forward runs on the 1,663-row held-out set.
  • It creates 23 complete periods and 1,380 out-of-sample daily returns on matching dates for both portfolio strategies.
  • The covered dates are 2017-05-23 through 2022-11-11; the final 31 held-out observations do not form another complete 60-row test window and are unused.
  • Predictions and fitted populations are saved and reload successfully.
  • HRP uses CVaR rather than the paper’s default risk measure.
  • Internal cv=None produces default five-fold K-fold, matching the paper’s omitted/default setting but remaining a non-temporal caveat.

QuantBrain: wrong outer scope and blocked runner

  • The code evaluates all 8,312 rows rather than only the 1,663-row held-out set, yielding 134 rather than 23 folds.
  • Internal CV is explicitly changed to three-fold K-fold.
  • A fold validator reports a false negative because it condemns intentional overlap between adjacent rolling training windows; all 134 actual folds still satisfy train_end < test_start.
  • The all-run command passes results_dir to a no-argument function and aborts with TypeError before Experiment 3.
  • Even after that signature fix, report-schema keys do not match the runner output.
5

Tests, dependencies, and automated review

Green tests do not answer the central semantic questions

Fresh test results and coverage gaps

  • Qwen: 34 + 51 + 39 = 124 current passes. The audit did not directly execute its top-level runner.
  • QuantBrain: 27/28 under the current unpinned environment; 28/28 under a compatible skfolio 0.6 environment. The current failure is exception-message drift.
  • Fifteen compatible Qwen Experiment 3 cases repeat expensive walk-forward work, including six full end-to-end runs; the suite is broad but inefficient.
  • Qwen tests never prove that test metrics are derived from X_test.
  • QuantBrain tests never call experiment entry points or the all-run command.
  • Neither repository has a complete lock. Qwen’s committed environment omits CVXPY/solver details; the current resolution is skfolio 0.20.1 with CVXPY 1.9.2, while the compatible reconstruction uses skfolio 0.6.0 with CVXPY 1.6.7. Combining skfolio 0.6.0 with the newest resolved CVXPY 1.9.2 fails in this reconstruction. Bundled dataset bytes are also unpinned.

Qwen automated disposition

Partially reproduced

Confidence: medium. The accessible judge correctly saw duplicate tables and a contradictory prior flag, but did not inspect source or execute code.

Visible root usage comprises 39,074 prompt and 7,137 completion tokens. It excludes the external Qwen repository job and PDF analysis, and the trace records no complete cost.

46,211visible QCA/GPT-5.4 tokens
26.8sfinal-review duration
12result/report files read
0source files or tests executed

What the accessible trace establishes

  • Requested model: runpod/qwen3.5-397b-a17b.
  • External job job-qca-runner-p8p92jz reported Succeeded.
  • Full root workflow: 61m 59.668s; external job: 58m 08.411s.
  • The judge was reasonably blind to model identity in its rendered message.
  • Visible usage includes 10,240 cache-read tokens but excludes Qwen repository generation and PDF-tool usage; complete cost is unrecoverable.

What the judge did not inspect

Source code
Tests and runner
Environment lock
Solver logs
Saved optimizer objects
Fold schedule
Predictions and weights
Plots through visual review
6

Direct article-two comparison

Qualified artifact-level judgments

DimensionStronger inspected artifactWhy
Recoverable evidenceQwen-attributed resultPersisted predictions, objects, tables, and plots
Experiment 1 evaluation structureQuantBrain-attributed resultSeparate train/test predictions rather than duplicate outputs
Experiment 1 conclusion integrityNeitherQwen does not test out of sample; QuantBrain reverses its actual negative result
Experiment 2 evidenceQwen-attributed resultPrior and loadings are recoverable despite a broken validation flag
Experiment 3 article fidelityQwen-attributed resultCorrect outer held-out scope and 23 walk-forward periods
End-to-end runnerQwen, qualifiedQuantBrain’s runner aborts; Qwen’s expensive root workflow was not independently rerun
Complete reproductionNeitherBoth retain method, verification, and environment gaps

The Qwen artifact is the more recoverable basis for continued work. QuantBrain’s separate train/test structure in Experiment 1 is a real strength, but its report, runner, and missing evidence require substantial repair.

The supplied QuantBrain run ID returns HTTP 404 under the supplied share token. Its exact task, requested model, job status, judge evidence, verdict, and usage cannot be verified. The paper trace retained an unversioned arXiv URL; version 2 was current long before the run and is highly likely, but the fetched PDF hash was not recorded by QCA. The generated task also did not freeze dataset bytes, solver, dependency versions, or an acceptance rule. Its generated narrative mixes the 2,263-row factor dataset with the 8,312-row S&P 500 workflows, another sign that summary text is not a reliable substitute for same-run artifacts.
Cross-cutting QCA findings

The recurring workflow weakness is independent verification.

Across both studies, QCA produced nontrivial but uneven repositories and structured outputs. The pipeline repeatedly allowed model-authored tests and summaries to stand in for external scientific verification.

05
Demonstrated

Nontrivial repository generation

Both studies contain multi-file scientific implementations, tests, reports, and machine-readable artifacts. Several workflows execute successfully.

Control failure

Failed jobs can still be judged

The 17 July failed QuantBrain job was followed by a judge reading the live default branch and returning the same verdict at higher stated confidence.

Oracle failure

Self-authored tests repeat misconceptions

One suite asserts the wrong spread property; another never verifies that “test” metrics use test data; reusable defects remain unimported.

Evidence selection

Judges mostly read generated summaries

Neither accessible final judge inspected source or executed tests. Adaptive file selection missed defects that direct source and numerical checks exposed.

ControlObserved across the supplied studiesRequired condition
Task identityMissing comparator traces and unequal visible scopesHash-identical PDF, experiment JSON, presets, limits, and prompts
Repository identityLive branches, absent generated SHAs, and one unrelated-history mergeEmpty repository per run and immutable generated commit captured by orchestration
Job gatingReview can continue after external coding failureHard stop unless generation succeeds and the expected commit exists
Scientific verificationModel-authored tests and narratives dominate the evidenceFixed external semantic tests plus independent metric recomputation
Judge evidenceAdaptive report reads; no source executionPredetermined evidence manifest with source, logs, outputs, environment, and verifier results
Usage and costVisible GPT-5.4/QCA tokens exclude tested-model generation and some PDF workPer-stage model, token, cache, retry, latency, and price records
Result semanticsOne coarse “partially reproduced” label hides different failure modesSeparate scores for completion, method fidelity, numerical result, evidence integrity, and test adequacy
Recommended next evaluation

A clean, comparable rerun protocol

The goal is to retain the useful repository-generation and paired-benchmark exercises while making every model response, execution result, and judgment independently reconstructible.

06

Freeze every input

Pin exact article, task, rubric, and dataset bytes; package/solver images; prompts, presets, tool permissions, model revisions, judges, seeds, and limits.

Use identical clean contexts

Send every model the hash-identical task in a clean repository and clean conversation context.

Persist the raw exchange

Store every tested-model response, extracted code, usage field, finish reason, retry, tool call, execution log, timestamp, and server-returned identity.

Gate terminal state

Abort evaluation on failed generation, missing commit, dirty output, or mismatched task hash. Never fall through to an existing default branch.

Run external semantic checks

Test formulas, stochastic process, calibration, train/test provenance, constraint satisfaction, fold geometry, and report-to-file consistency.

Recompute headline numbers

Calculate metrics from persisted raw outputs with independent code. Require every narrative claim to resolve to the same-run evidence.

Judge blind on fixed evidence

Supply the same predetermined manifest to pinned reviewers, adjudicate disagreements, correct for planned multiplicity, and keep unlike score families separate.

Report dimensions separately

Separate artifact completion, article fidelity, numerical behavior, evidence integrity, and test adequacy instead of collapsing them into one verdict.

Comprehensive evidence index

Primary sources and complete audit records

This page synthesizes the principal findings from both QCA article-reproduction investigations. The linked Markdown and JSON records preserve exact commands, hashes, environments, file-level references, metrics, and verification limits.

07
A

Avellaneda–Stoikov evidence

Readable auditExecutive synopsis, comparison, interpretation, and follow-up.Open audit →
Verification logEnvironment, tests, fresh numerical probes, hashes, and link checks.Open verification →
Provenance inventoryDeveloper statements, commits, paper versions, trace IDs, and earlier runs.Open sources →
Machine-readable evidenceStructured repository, numerical, trace, environment, and conclusion records.Open JSON →
Uploaded benchmark paper14-page preprint used by the QCA task.Open PDF →
Later published paperEight-page NYU-hosted scan with materially different tables.Open PDF →
Corrected QuantBrain traceRequested model, reported job status, and GPT-5.4 review.Open trace →
Missing Qwen run lookupDirect public API request returns HTTP 404 under the supplied share.Open lookup →
B

skfolio evidence

Readable auditExecutive synopsis and all three experiment assessments.Open audit →
Repository and numerical auditSource-level defects, reruns, tests, dependencies, and comparative assessment.Open detailed findings →
Verification logPinned checkouts, fresh execution, corrected predictions, paper capture, and completion checks.Open verification →
Provenance inventoryDeveloper attribution, commits, paper versions, and trace IDs.Open sources →
Machine-readable evidenceStructured commits, outputs, metrics, tests, trace usage, and conclusions.Open JSON →
Reference paperskfolio: Portfolio Optimization in Python; arXiv v2 highly likely, trace did not pin bytes.Open paper record →
Accessible Qwen traceQCA orchestration, reported job status, and automated judgment.Open trace →
Supplied QuantBrain trace URLThe human-facing URL supplied for provenance is unavailable within its public share.Open supplied URL →
Missing QuantBrain run lookupDirect public API request returns HTTP 404 under the supplied share.Open lookup →