Dataset Viewer
The dataset could not be loaded because the splits use different data file formats, which is not supported. Read more about the splits configuration. Click for more details.
Couldn't infer the same data file format for all splits. Got {NamedSplit('train'): ('parquet', {}), NamedSplit('validation'): ('json', {}), NamedSplit('test'): ('json', {})}
Error code: FileFormatMismatchBetweenSplitsError
Need help to make the dataset viewer work? Make sure to review how to configure the dataset viewer, and open a discussion for direct support.
π¦ sharky-workbench
Build logs, audits, sha-pinned scripts, and eval reports for the sharky project (model: luispoveda93/sharky-0.5B). Renamed from minicpm4-pcap-kev-workbench (contents identical).
π€ Want to help improve the model? Start with phase9/CONTRIBUTING.md β seven concrete open tasks, from a $0.10 CPU render to an $11 retrain, with the exact commands and the gotchas that cost us hours.
Where the project stands (TL;DR)
- In-distribution: locked-split 0.9865 accuracy, calibrated, gate 40% auto-close @ 99.9% precision.
- Out-of-distribution (phase8): collapses β CIC-IDS2017 zero-shot 0.27 acc, real-malware recall 7.4%, 34% of benign 2018 traffic false-flagged.
- Phase9: two retrain candidates (v3, v3b) executed under pre-registered gates and rejected β v3 fixed real-malware recall (7.4%β99%) and unseen-benign (58%β97%) but not the CIC-2017 benign boundary.
- Diagnosed: benign-web shortcut β benign web-destination flows false-flagged 85%, benign DNS 0%; UNSW training benign is 4.4% web vs CIC-2017's 46%. The v4 recipe (benign-web oversampled retrain) is specified in
phase9/README.md.
Key artifacts
| Path | What it is |
|---|---|
phase2/build_summary_v3.json |
v1 states build summary (49,745 states, verified hex rendering) |
phase2/unsw15_label_audit.json |
Full per-file label histogram of the UNSW-NB15 Packet-Bytes view (18 files, ~180M packets) |
phase3a/train_3a.py |
sha-pinned v1 training script (the validated MiniCPM shim lives here) |
phase4/eval_report.json |
Phase 4 eval: locked-split 0.980, calibration, gate curves for v1 |
phase5/validation_v4.json |
Demo Space end-to-end validation report |
phase6/ood_diag_v1.json |
v1 OOD diagnosis (held-out UNSW files F6+F16) |
phase6/eval_v1v2.json |
Head-to-head v1 vs v2 β v2 wins all sets (locked 0.9865, UNSW OOD 0.9905, DAPT OOD 0.9848 vs v1 mean 0.801) |
phase6/train_v2.py |
sha-pinned v2 warm-start training script |
phase6/build_v2_dapt.py |
sha-pinned DAPT-2020 ingestion script (flat-key 5-tuple CSV matching) |
phase7/benchmark_plan.md |
Benchmark plan of record (rev 2) β kev-referenced, all-local: kev-0.5b/0.6b/4b pcap head-to-heads via kev.serve, untrained-head control, optional Laya, shared public suites, gate coverage@precision curves. Direct Jev API head-to-head deferred until access lands (protocol preserved in appendix; kev-0.6b's published head-to-head serves as the indirect Jev anchor) |
phase7/eval_bootstrap_p0.json + manifests.json |
P0 protocol hardening (executed) β bootstrap 95% CIs on all six checkpoint-set pairs from confusion counts, frozen suite manifests (incl. documented non-identical rename of train_3a.py) |
phase7/eval_pcap_headtohead.json |
Head-to-head aggregates (executed) β sharky 0.9865/0.9848 vs untrained-head control 0.4350/0.4543 and zero-shot kev-0.5b/0.6b 0.4007/0.4584 (constant-benign collapse); kev-4b completed in phase8 (0.4248 locked / 0.4548 DAPT, same collapse shape) |
phase7/leaderboard.md + postprocess_p7.json |
Leaderboard + post-processing (executed) β paired bootstrap deltas (all baselines 0.53β0.59 below sharky, CIs exclude 0), sharky gate curves (auto-close 40.1% @ 99.9% precision locked; 45.8% max coverage at β€5% error DAPT), kev probability sanity |
phase7/temp_refit.json |
Temperature re-audit β refit on the exact v2 calibration partition lands on T=1.7411, zero drift vs card (8,420 = 2,105 records Γ 4 questions) |
phase7/eval_headtohead.py + postprocess_p7.py |
sha-pinned phase7 harness (per-arm upserts) and CPU post-processor |
phase7/perrecord_minicpm.json / perrecord_kev.json |
Per-record raw verdict probabilities, ids and labels for every executed arm (paired-bootstrap-ready) |
phase8/benchmark_plan.md |
phase8 plan of record (rev 2) β up-to-date OOD suites: S1 CIC-IDS2017 zero-shot, S2 CSE-CIC-IDS2018, S3 CICIoT2023 (blocked: empty mirror), S4 CTU-13, C0 kev-4b carry-over |
phase8/leaderboard.md |
phase8 leaderboard (final) β OOD collapse numbers per suite with per-family breakdown, gate curves, kev-4b row filled |
phase8/suites/ |
Eval suites: states_cicids2017.parquet (10,200 states, 7 families), states_ctu13_botnet42.parquet (1,200 real-malware), states_cicids2018_d1.parquet (2,000 benign 2018) |
phase8/eval_*.json + perrecord_s1.json / perrecord_s4.json |
Per-suite results + per-record raw probabilities (calibration/bootstrap-ready) |
phase8/eval_s1.py / eval_s4.py / render_*.py |
sha-pinned phase8 harnesses: suite renderers (UNSW-identical byte format), 2-mode eval (MiniCPM shim / kev-native) |
phase9/benchmark_plan.md |
phase9 plan of record β v3 mixed-corpus retrain under a pre-registered 5-gate promotion contract |
phase9/gate_v3.json / gate_v3b.json |
Gate batteries (executed) β v3: 3/5 (Neris 7.4%β99%, unseen-benign 58%β97%, locked 0.9808; S1 mF1 0.706); v3b: 4/7 but benign recall 0.36β0.24 (wrong direction) |
phase9/REJECTED.md / REJECTED_v3b.md |
Rejection records β per-gate deltas, single failure mode, A2 decision ladder |
phase9/train_v3.py / train_v3b.py / render_train_*.py |
sha-pinned phase9 trainers (warm-start + from-base recipes) and eval-isolated training-pool renderers |
phase9/diag_s1_benign.json |
Diagnosis (executed) β benign-web shortcut: web-dst benign 85% false-flag, dns 0%, payload-65-256B 0%; UNSW benign 4.4% web vs CIC-17 46% |
phase9/CONTRIBUTING.md |
Contributor guide β 7 open tasks (T1βT7), runnable commands, gotchas, gate discipline |
Conventions
- Every script that ever ran on GPU is published here and fetched by jobs at a pinned commit with in-job sha256 verification β the byte-exactness protocol used throughout the project.
- Audits and summaries are persisted here so results survive job loss (per-arm/per-file checkpointing pattern).
- Gate discipline: training arms pre-register a gate table, run, and publish pass or fail. Two of our own retrains are published rejections β that's the standard, not an exception.
- Contributing: tasks, commands, and compute notes live in
phase9/CONTRIBUTING.md. PRs and discussions welcome on this repo.
All workbench content is reproducible provenance for luispoveda93/sharky-0.5B and luispoveda93/sharky-triage-states.
- Downloads last month
- 756