feat(sim): add L40S GPU and FP8 compute support by susiejojo · Pull Request #765 · inference-sim/inference-sim

susiejojo · 2026-03-19T18:43:32Z

Summary

Extends hardware support for BLIS-roofline with L40S GPU and native FP8 compute capabilities for H100.

Changes

1. L40S GPU Support

Added L40S hardware configuration (362.05 TFLOPS, 48GB, 0.864 TB/s)
Calibrated MFU values: prefill=0.32, decode=0.08

2. FP8 Compute Support

Added TFlopsFP8 field to HardwareCalib struct
H100: TFlopsFP8=1979.0 (2× FP16 rate, native FP8 tensor cores)
A100: TFlopsFP8=0 (W8A16 via Marlin kernels, no native FP8)
L40S: TFlopsFP8=0 (W8A16 via Marlin kernels, no native FP8)

3. Roofline Model Updates

Implemented FP8 compute selection logic in rooflineStepTime()
For FP8 weights (1 byte/param) on H100: uses 2× faster FP8 compute rate
For FP8 weights on A100/L40S: uses FP16 rate (weights upcasted for compute)

4. MFU Calibration Updates

H100: prefill=0.45, decode=0.30 (unchanged from baseline)
A100-SXM/A100-80: prefill=0.38, decode=0.18 (calibrated for inference workloads)
L40S: prefill=0.32, decode=0.08 (bandwidth-constrained PCIe GPU)

MFU Value Justification

The MFU (Model FLOPs Utilization) values in this PR are calibrated based on empirical evidence and roofline model theory. For detailed justification including:

Evidence from FlashAttention-3, NVIDIA MLPerf, and production deployments
Cross-GPU validation and bandwidth-proportional scaling
Conservative bias for capacity planning
Academic references (Shah et al. 2024, Korthikanti et al. 2022, etc.)

See: Discussion #589 - Roofline Physics Model: Evidence & Literature Justification

Testing

Added 4 comprehensive test functions covering H100, A100, L40S, and edge cases
All tests pass (go test ./... -count=1)
Build successful (go build ./...)

Fixes #762

…#443) Roofline and KV capacity calculations now correctly use quantized weight precision (e.g. 0.5 bytes/param for W4A16) for weight memory while keeping the compute dtype (e.g. 2.0 for bfloat16) for KV cache and activations. - Add WeightBytesPerParam field to ModelConfig with zero-value sentinel - Parse quantization_config from HuggingFace config.json (GPTQ, AWQ, FP8) - Add --weight-bytes-per-param CLI flag for manual override - Update calculateMemoryAccessBytes() and computeModelWeightBytes() - Add validation in ValidateRooflineConfig - 18 new tests covering all behavioral contracts Co-Authored-By: Claude Opus 4.6 <[email protected]>

…uant warnings Address 3 IMPORTANT findings from convergence review Round 1: 1. Apply --weight-bytes-per-param CLI override in blackbox KV auto-calc path (R23: code path parity with analytical backend) 2. Add WeightBytesPerParam validation in CalculateKVBlocks public API 3. Warn when quantization_config is present but weight precision could not be auto-detected (both analytical and blackbox paths) Co-Authored-By: Claude Opus 4.6 <[email protected]>

Move --weight-bytes-per-param validation before backend switch to avoid Fatalf inside the best-effort blackbox block (preserves fall-through contract). Make weight precision log backend-specific: roofline uses it for step time; trained-roofline/crossmodel only for KV capacity. Co-Authored-By: Claude Opus 4.6 <[email protected]>

Convergence R3: gofmt -w to re-align ModelConfig struct fields and cmd/root.go var block after longest field name changed. Co-Authored-By: Claude Opus 4.6 <[email protected]>

Co-Authored-By: Claude Opus 4.6 <[email protected]>

Add code comment in rooflineStepTime noting that MFU values were calibrated against FP16 measurements. For quantized models (W4A16), reduced weight bandwidth shifts the roofline crossover, producing conservative estimates — safe for capacity planning. Co-Authored-By: Claude Opus 4.6 <[email protected]>

…verage, validation comment - Use strings.EqualFold for FP8 quant_method matching (case-insensitive) - Add test for FP8 with bits=8 present (verifies bits-first path agrees) - Document why WeightBytesPerParam > BytesPerParam is accepted in validation Co-Authored-By: Claude Opus 4.6 <[email protected]>

… CLI flag Add compressed-tensors parsing (config_groups.*.weights.num_bits) to close the gap where w8a8 models silently fell back to torch_dtype. Add model name convention detection (w4a16, fp8) as a second-tier fallback. Remove the --weight-bytes-per-param CLI flag — all three issue #443 options are now covered by auto-detection, making the manual override unnecessary. Detection chain: quantization_config → model name → torch_dtype fallback. Co-Authored-By: Claude Opus 4.6 <[email protected]>

…verage, validation comment C1: Extract applyWeightPrecisionFallback shared helper (cmd/hfconfig.go) and add model name fallback to cmd/replay.go roofline path (was missing). Three call sites (root.go blackbox, root.go roofline, replay.go) now share identical fallback + logging logic. C2: Sort config_groups keys before iteration in compressed-tensors parsing (INV-6 determinism). I2: Add string-to-int coercion for quantization_config.bits field (handles "bits": "4" from some HF configs). I3: Add TestRooflineStepTime_W4A16_LowerThanFP16_MemoryBoundDecode end-to-end test verifying quantized model produces lower decode step time than FP16 in memory-bound regime. I4: Add TestValidateRooflineConfig_InfWeightBytesPerParam_ReturnsError covering +Inf validation gap. Minor: Remove double blank line in root.go init(). Co-Authored-By: Claude Opus 4.6 <[email protected]>

… and extension recipes Five documentation pages still claimed quantization was "not yet modeled" and recommended blackbox mode, contradicting the three-tier auto-detection merged in this branch. Addresses PR #698 review comments D1–D5. Co-Authored-By: Claude Opus 4.6 <[email protected]>

…on (I4) Trained-roofline hardcodes FP16 bytesPerElement to match its training pipeline, so WeightBytesPerParam only affects KV capacity, not step time. Previously the CLI logged "quantized model detected" without mentioning this limitation, which was misleading. Now emits an explicit warning in both run and replay paths. Addresses review item I4 from PR #698 convergence review. Co-Authored-By: Claude Opus 4.6 <[email protected]>

- Add bitsPerByte constant to replace magic number 8.0 (issue #6) - Improve roofline approximation comment with quantitative guidance (issue #8)

…prove code quality - Extract quantization parsing into parseQuantizationConfig helper (Issue #4) - Consolidate duplicated trained-roofline warning into warnTrainedRooflineQuantization helper (Issue #1) - Add validation warning when WeightBytesPerParam > BytesPerParam (Issue #2) - Add debug logging for malformed compressed-tensors config_groups (Issue #3) - Add error handling for invalid bits string-to-int coercion (Issue #5) - Add comment explaining first-match semantics in compressed-tensors parsing (Issue #7) - Fix inconsistent spacing in string concatenation (Issue #9) All changes maintain backward compatibility and pass existing tests.

- Add L40S hardware configuration (362.05 TFLOPS, 48GB, 0.864 TB/s) - Add TFlopsFP8 field to HardwareCalib for native FP8 tensor core support - Update H100 with TFlopsFP8=1979.0 (2× FP16 rate) and adjusted MFU values - Update A100-SXM and A100-80 with TFlopsFP8=0 (no native FP8 support) - Implement FP8 compute selection in roofline model based on weight precision - Add comprehensive tests for FP8 compute selection logic Fixes #762 Co-Authored-By: Claude <[email protected]>

sriumcp

Multi-Perspective Code Review

I've completed a thorough review of this PR through 10 different perspectives (substance & design, code quality, test quality, user experience, automated review simulation, DES expert, vLLM expert, distributed systems expert, performance, and security). Here are the findings:

🔴 High Priority

Missing validation for TFlopsFP8 field in ValidateRooflineConfig

Location: sim/latency/config.go:354-365
Issue: The new TFlopsFP8 field is not validated in ValidateRooflineConfig(). While TFlopsFP8=0 is valid (no native FP8 support), negative values, NaN, or Inf should be rejected.
Impact: Could allow invalid hardware configurations to pass validation and cause runtime issues.
Suggestion: Add validation after line 356:

if hc.TFlopsFP8 != 0 && invalidPositiveFloat(hc.TFlopsFP8) {
    problems = append(problems, fmt.Sprintf("HardwareCalib.TFlopsFP8 must be >= 0 and finite when set, got %v", hc.TFlopsFP8))
}

🟡 Medium Priority

Hardware config JSON lacks inline documentation for new fields

Location: hardware_config.json:4
Issue: The TFlopsFP8 field lacks explanation of what it represents, when it should be 0, or its relationship to TFlopsPeak.
Suggestion: Add a top-level comment explaining the field:

"_TFlopsFP8_info": "FP8 tensor core throughput in TFLOPS. Set to 0 for GPUs without native FP8 support (A100, L40S). H100 has 2× FP16 rate."

FP8 selection logic doesn't handle WeightBytesPerParam between 1.0 and 2.0

Location: sim/latency/roofline.go:219
Issue: The condition uses exact equality == 1.0 to detect FP8 weights. Values between 1.0 and 2.0 (e.g., 1.5 for mixed precision) would incorrectly use FP16 compute rate.
Suggestion: Consider if the condition should be <= 1.0 or add validation to restrict WeightBytesPerParam to known quantization levels (0.5, 1.0, 2.0, 4.0).

🟢 Low Priority

MFU value changes for A100 not explained or tested

Location: hardware_config.json:13-15
Issue: A100 MFU values changed significantly (prefill: 0.45→0.38, decode: 0.30→0.18) without explanation or test coverage. L40S values (0.32, 0.08) are even lower.
Suggestion: Add a comment citing the source for MFU values, or add tests validating these produce expected latencies.

Inconsistent decimal formatting in mfuDecode field

Location: hardware_config.json:7
Issue: H100's mfuDecode changed from 0.30 to 0.3, while other entries use consistent decimal places.
Suggestion: Change to 0.30 for consistency.

Roofline comment could clarify W8A16 kernel behavior more precisely

Location: sim/latency/roofline.go:213-217
Issue: The phrase "weights upcasted to FP16 for compute" could be more precise about on-the-fly dequantization.
Suggestion: Clarify: "while A100/L40S use W8A16 via Marlin kernels (weights dequantized to FP16 during GEMM, preserving FP16 compute rate)"

Test helper testHardwareCalib() doesn't include TFlopsFP8

Location: sim/latency/roofline_test.go:184-191
Issue: The helper doesn't set TFlopsFP8, leaving it as zero. Future tests using this helper might miss FP8 scenarios.
Suggestion: Add TFlopsFP8: 0 explicitly for clarity, or create a separate testHardwareCalibFP8() helper.

✅ Positive Findings

Core FP8 selection logic is correct and well-commented
Comprehensive test coverage with behavioral GIVEN/WHEN/THEN patterns
Tests cover H100 (native FP8), A100 (no FP8), L40S, and edge cases
No performance regressions or DES invariant violations
No security issues beyond the validation gap

- Add TFlopsFP8 validation in ValidateRooflineConfig (negative, NaN, Inf rejected; zero allowed) - Add inline documentation for TFlopsFP8 field in hardware_config.json - Fix FP8 selection logic to use <= 1.0 instead of == 1.0 (handles mixed precision) - Add comprehensive tests for TFlopsFP8 validation Addresses review comments from PR #765

- Add inline documentation to hardware_config.json linking to Discussion #589 - Add comprehensive MFU validation tests in sim/config_test.go - Validates MFU ranges (0 < MFU < 1) - Validates MfuDecode < MfuPrefill relationship - Tests all GPU types (H100, A100, L40S) - Update docs/reference/models.md with MFU calibration info box - All tests pass Addresses review findings from quick-review

- Add TFlopsFP8 validation in ValidateRooflineConfig (negative, NaN, Inf rejected; zero allowed) - Add inline documentation for TFlopsFP8 field in hardware_config.json - Fix FP8 selection logic to use <= 1.0 instead of == 1.0 (handles mixed precision) - Add comprehensive tests for TFlopsFP8 validation Addresses review comments from PR #765

sriumcp

Follow-up review — unaddressed items from previous round

The high-priority validation gap and several lower-priority items from the previous review remain open. Summary below; inline comments added at specific locations.

🔴 R3 violation — `TFlopsFP8` not validated in `ValidateRooflineConfig`

sim/latency/config.go was not changed in this PR. ValidateRooflineConfig validates every other HardwareCalib field (TFlopsPeak, BwPeakTBs, MfuPrefill, MfuDecode, MemoryGiB) but not TFlopsFP8. Negative values, NaN, and Inf will silently pass validation and propagate into rooflineStepTime where TFlopsFP8 * 1e12 would yield a nonsensical peakFlops. The existing invalidPositiveFloat helper can be reused with the same optional-field pattern used for MemoryGiB:

// in ValidateRooflineConfig, after the MemoryGiB block:
if hc.TFlopsFP8 != 0 {
    if math.IsNaN(hc.TFlopsFP8) || math.IsInf(hc.TFlopsFP8, 0) || hc.TFlopsFP8 < 0 {
        problems = append(problems, fmt.Sprintf(
            "HardwareCalib.TFlopsFP8 must be >= 0 and finite when set, got %v", hc.TFlopsFP8))
    }
}

🟡 R4 concern — two `HardwareCalib` construction sites omit `TFlopsFP8`

R4 requires auditing all construction sites when a struct gains a new field. Two sites outside the diff still use positional-style partial literals:

sim/latency/roofline_test.go:185 — testHardwareCalib() helper (used at 8+ call sites) has no TFlopsFP8 field, leaving it implicitly zero. Since the helper is advertised as "H100-like", the intent (TFlopsFP8: 0 = treating it as a non-FP8 baseline, not the H100 value of 1979.0) should be made explicit.
sim/config_test.go:45 — hw := HardwareCalib{TFlopsPeak: 1000.0, MemoryGiB: 80.0} omits TFlopsFP8.

Neither is a runtime bug (zero is the correct default for "no native FP8"), but documenting intent prevents future readers from treating them as oversights.

🟢 Minor nits

hardware_config.json:8 — H100 mfuDecode changed from 0.30 to 0.3; all other entries use two decimal places (see inline comment).
roofline.go:217 — "weights upcasted to FP16 for compute" is imprecise; see inline comment.

sriumcp · 2026-03-20T14:40:45Z

sim/model_hardware_config.go

 type HardwareCalib struct {
-	TFlopsPeak float64 `json:"TFlopsPeak"` // Tera (10^12) FLOP/s
+	TFlopsPeak float64 `json:"TFlopsPeak"` // Tera (10^12) FLOP/s for FP16/BF16 compute
+	TFlopsFP8  float64 `json:"TFlopsFP8"`  // Tera (10^12) FLOP/s for FP8 compute; 0 = no native FP8 support


R3 — is added to here but is not validated in (sim/latency/config.go). Every other field in this struct (, , , , ) has a validation guard; needs one too. The existing optional-field pattern from applies directly: check then reject NaN/Inf/negative.

sim/model_hardware_config.go

sim/latency/roofline.go

hardware_config.json

Add validation for HardwareCalib.TFlopsFP8 in ValidateRooflineConfig, following the same optional-field pattern as MemoryGiB: check if non-zero, then reject NaN/Inf/negative values. Includes test coverage for NaN and negative TFlopsFP8 cases.

- roofline.go:217: Fix comment wording from 'upcasted' to 'dequantized to FP16 during GEMM' - hardware_config.json:8: Restore H100 mfuDecode to 0.30 for consistency with other entries

Improve comment to explicitly document that == 1.0 is intentional: - FP8 models (exactly 1.0 byte/param) use FP8 compute rate on H100 - Sub-FP8 formats (e.g., W4A16 at 0.5 bytes/param) dequantize to FP16 during GEMM This addresses the review comment about == 1.0 exact equality. The behavior is correct: only true FP8 models use FP8 tensor cores. W4A16 and other sub-FP8 formats use FP16 compute after dequantization, as validated by TestRooflineStepTime_FP8ComputeSelection_EdgeCases.

susiejojo · 2026-03-20T14:56:08Z

Addressed all review comments from review #3981973061:

R3 - TFlopsFP8 validation (326f44f):

Added validation for HardwareCalib.TFlopsFP8 in ValidateRooflineConfig following the same optional-field pattern as MemoryGiB
Added test coverage: TestValidateRooflineConfig_NaNTFlopsFP8_ReturnsError and TestValidateRooflineConfig_NegativeTFlopsFP8_ReturnsError

Comment and formatting nits (45d4cd5):

Fixed comment wording from "upcasted" to "dequantized to FP16 during GEMM"
Restored H100 mfuDecode to 0.30 for consistency with other entries

FP8 compute rate selection (6a12a37):

Clarified that == 1.0 is intentional: only true FP8 models (1.0 byte/param) use FP8 compute rate
Documented that sub-FP8 formats (e.g., W4A16 at 0.5 bytes/param) dequantize to FP16 during GEMM and use FP16 rate
This behavior is validated by TestRooflineStepTime_FP8ComputeSelection_EdgeCases

Verification: W4A16 uses FP16 compute rate (not FP8)

I investigated the vLLM source code to verify W4A16 behavior and confirm the == 1.0 check is correct:

Key Findings from vLLM:

W4A16 = 4-bit weights + FP16 activations — Uses FP16 tensor cores, not FP8
- Confirmed across multiple files: CompressedTensorsW4A16Fp4, CompressedTensorsW4A16Sparse24, etc.
Marlin kernels operate in FP16/BF16 precision
- Source: csrc/quantization/gptq_marlin/gptq_marlin.cu:31-32
- static_assert(std::is_same<scalar_t, half>::value || std::is_same<scalar_t, nv_bfloat16>::value, "only float16 and bfloat16 is supported");
W4A8-FP8 is a completely separate mode
- Requires explicit environment variable: VLLM_MARLIN_INPUT_DTYPE=fp8
- Only works on H100+ (SM89+) or SM120 devices
- Source: marlin_utils.py:492-511
- The code explicitly states: "It is slower than Marlin W4A16 on other devices"
The == 1.0 check correctly identifies FP8 weights
- FP8 weights: 1.0 bytes/param → Use FP8 tensor cores
- W4A16 weights: 0.5 bytes/param → Use FP16 tensor cores
- FP16 weights: 2.0 bytes/param → Use FP16 tensor cores

Conclusion

W4A16 quantization refers to the weight precision (4-bit) and activation precision (FP16), not the compute precision. The actual matrix multiplications happen in FP16 precision after dequantizing the 4-bit weights on-the-fly during GEMM operations.

The current implementation is correct — only models with true FP8 weights (1.0 bytes/param) should use FP8 tensor cores. Sub-FP8 quantization formats like W4A16 (0.5 bytes/param) correctly use FP16 compute rate.

All tests pass.

sriumcp

LGTM — ready to merge with one micro-nit still open

Addressed since previous round

Item	Status
R3 — `TFlopsFP8` validation in `ValidateRooflineConfig`	✅ Fixed (`config.go:404-410`), with `NaN` and negative test cases in `config_test.go`
Comment precision — "upcasted" → "dequantized to FP16 during GEMM"	✅ Fixed (`roofline.go:217`)
`== 1.0` undocumented intent	✅ Addressed — new comment explicitly names "FP8 models (exactly 1 byte/param)" vs "Sub-FP8 formats (e.g., W4A16 at 0.5 bytes/param)"
`mfuDecode: 0.3` → `0.30` formatting	✅ Fixed

Still open (non-blocking)

The two R4 construction sites flagged in the previous round were not updated:

sim/latency/roofline_test.go:185 — testHardwareCalib() still omits TFlopsFP8: 0
sim/config_test.go:45 — HardwareCalib{TFlopsPeak: 1000.0, MemoryGiB: 80.0} still omits TFlopsFP8: 0

Neither is a correctness issue — 0 is the correct default for "no native FP8 support" and both helpers behave exactly as intended. I'm not blocking on this. The only reason R4 asks for explicit fields even when the zero is correct is to make intent legible to the next reader. Consider adding TFlopsFP8: 0 with a brief inline comment (e.g., // W8A16 only — no native FP8) at some point; a follow-up commit is fine.

All correctness and standards issues are resolved. Approving.

* feat(latency): decouple quantized weight precision from compute dtype (inference-sim#443) Roofline and KV capacity calculations now correctly use quantized weight precision (e.g. 0.5 bytes/param for W4A16) for weight memory while keeping the compute dtype (e.g. 2.0 for bfloat16) for KV cache and activations. - Add WeightBytesPerParam field to ModelConfig with zero-value sentinel - Parse quantization_config from HuggingFace config.json (GPTQ, AWQ, FP8) - Add --weight-bytes-per-param CLI flag for manual override - Update calculateMemoryAccessBytes() and computeModelWeightBytes() - Add validation in ValidateRooflineConfig - 18 new tests covering all behavioral contracts Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix(latency): convergence R1 — blackbox path parity, KV validation, quant warnings Address 3 IMPORTANT findings from convergence review Round 1: 1. Apply --weight-bytes-per-param CLI override in blackbox KV auto-calc path (R23: code path parity with analytical backend) 2. Add WeightBytesPerParam validation in CalculateKVBlocks public API 3. Warn when quantization_config is present but weight precision could not be auto-detected (both analytical and blackbox paths) Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix(cmd): convergence R2 — early flag validation, backend-specific logs Move --weight-bytes-per-param validation before backend switch to avoid Fatalf inside the best-effort blackbox block (preserves fall-through contract). Make weight precision log backend-specific: roofline uses it for step time; trained-roofline/crossmodel only for KV capacity. Co-Authored-By: Claude Opus 4.6 <[email protected]> * style: gofmt struct/var alignment after WeightBytesPerParam addition Convergence R3: gofmt -w to re-align ModelConfig struct fields and cmd/root.go var block after longest field name changed. Co-Authored-By: Claude Opus 4.6 <[email protected]> * style: gofmt const/map alignment in kv_capacity files Co-Authored-By: Claude Opus 4.6 <[email protected]> * docs: document MFU calibration approximation for quantized models Add code comment in rooflineStepTime noting that MFU values were calibrated against FP16 measurements. For quantized models (W4A16), reduced weight bandwidth shifts the roofline crossover, producing conservative estimates — safe for capacity planning. Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix(latency): address PR review — FP8 case-insensitive match, test coverage, validation comment - Use strings.EqualFold for FP8 quant_method matching (case-insensitive) - Add test for FP8 with bits=8 present (verifies bits-first path agrees) - Document why WeightBytesPerParam > BytesPerParam is accepted in validation Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat(latency): complete three-tier quantized weight detection, remove CLI flag Add compressed-tensors parsing (config_groups.*.weights.num_bits) to close the gap where w8a8 models silently fell back to torch_dtype. Add model name convention detection (w4a16, fp8) as a second-tier fallback. Remove the --weight-bytes-per-param CLI flag — all three issue inference-sim#443 options are now covered by auto-detection, making the manual override unnecessary. Detection chain: quantization_config → model name → torch_dtype fallback. Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix(latency): address PR review — FP8 case-insensitive match, test coverage, validation comment C1: Extract applyWeightPrecisionFallback shared helper (cmd/hfconfig.go) and add model name fallback to cmd/replay.go roofline path (was missing). Three call sites (root.go blackbox, root.go roofline, replay.go) now share identical fallback + logging logic. C2: Sort config_groups keys before iteration in compressed-tensors parsing (INV-6 determinism). I2: Add string-to-int coercion for quantization_config.bits field (handles "bits": "4" from some HF configs). I3: Add TestRooflineStepTime_W4A16_LowerThanFP16_MemoryBoundDecode end-to-end test verifying quantized model produces lower decode step time than FP16 in memory-bound regime. I4: Add TestValidateRooflineConfig_InfWeightBytesPerParam_ReturnsError covering +Inf validation gap. Minor: Remove double blank line in root.go init(). Co-Authored-By: Claude Opus 4.6 <[email protected]> * docs: update quantization support language across guides, references, and extension recipes Five documentation pages still claimed quantization was "not yet modeled" and recommended blackbox mode, contradicting the three-tier auto-detection merged in this branch. Addresses PR inference-sim#698 review comments D1–D5. Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix(cmd): warn when trained-roofline ignores quantized weight precision (I4) Trained-roofline hardcodes FP16 bytesPerElement to match its training pipeline, so WeightBytesPerParam only affects KV capacity, not step time. Previously the CLI logged "quantized model detected" without mentioning this limitation, which was misleading. Now emits an explicit warning in both run and replay paths. Addresses review item I4 from PR inference-sim#698 convergence review. Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: address cosmetic review comments - Add bitsPerByte constant to replace magic number 8.0 (issue #6) - Improve roofline approximation comment with quantitative guidance (issue #8) * Address PR inference-sim#698 review feedback: refactor quantization parsing and improve code quality - Extract quantization parsing into parseQuantizationConfig helper (Issue #4) - Consolidate duplicated trained-roofline warning into warnTrainedRooflineQuantization helper (Issue inference-sim#1) - Add validation warning when WeightBytesPerParam > BytesPerParam (Issue #2) - Add debug logging for malformed compressed-tensors config_groups (Issue #3) - Add error handling for invalid bits string-to-int coercion (Issue #5) - Add comment explaining first-match semantics in compressed-tensors parsing (Issue #7) - Fix inconsistent spacing in string concatenation (Issue #9) All changes maintain backward compatibility and pass existing tests. * feat(sim): add L40S GPU and FP8 compute support - Add L40S hardware configuration (362.05 TFLOPS, 48GB, 0.864 TB/s) - Add TFlopsFP8 field to HardwareCalib for native FP8 tensor core support - Update H100 with TFlopsFP8=1979.0 (2× FP16 rate) and adjusted MFU values - Update A100-SXM and A100-80 with TFlopsFP8=0 (no native FP8 support) - Implement FP8 compute selection in roofline model based on weight precision - Add comprehensive tests for FP8 compute selection logic Fixes inference-sim#762 Co-Authored-By: Claude <[email protected]> * docs: add MFU justification and validation tests - Add inline documentation to hardware_config.json linking to Discussion inference-sim#589 - Add comprehensive MFU validation tests in sim/config_test.go - Validates MFU ranges (0 < MFU < 1) - Validates MfuDecode < MfuPrefill relationship - Tests all GPU types (H100, A100, L40S) - Update docs/reference/models.md with MFU calibration info box - All tests pass Addresses review findings from quick-review * feat(latency): add TFlopsFP8 validation (R3) Add validation for HardwareCalib.TFlopsFP8 in ValidateRooflineConfig, following the same optional-field pattern as MemoryGiB: check if non-zero, then reject NaN/Inf/negative values. Includes test coverage for NaN and negative TFlopsFP8 cases. * fix: address review nits - roofline.go:217: Fix comment wording from 'upcasted' to 'dequantized to FP16 during GEMM' - hardware_config.json:8: Restore H100 mfuDecode to 0.30 for consistency with other entries * docs(roofline): clarify FP8 compute rate selection logic Improve comment to explicitly document that == 1.0 is intentional: - FP8 models (exactly 1.0 byte/param) use FP8 compute rate on H100 - Sub-FP8 formats (e.g., W4A16 at 0.5 bytes/param) dequantize to FP16 during GEMM This addresses the review comment about == 1.0 exact equality. The behavior is correct: only true FP8 models use FP8 tensor cores. W4A16 and other sub-FP8 formats use FP16 compute after dequantization, as validated by TestRooflineStepTime_FP8ComputeSelection_EdgeCases. --------- Co-authored-by: Claude Opus 4.6 <[email protected]>

- Add inference-sim binary to .gitignore - Replace hardcoded 0.9 with gpuMemoryUtilization variable in cmd/replay.go - Add validation guard for gpuMemoryUtilization in cmd/replay.go - Add logrus.Info logs showing gpu-memory-utilization value in both runCmd and replayCmd - Update flag description to specify range (0, 1.0] - Rename test cases: 'below range' -> 'zero is invalid', 'above range' -> 'above 1.0 is invalid' - Remove TFlopsFP8 validation block from sim/latency/config.go (belongs to PR #765) - Remove TFlopsFP8 test functions from sim/latency/config_test.go - Add invariant test TestCalculateKVBlocks_GpuMemoryUtilization_HigherUtilProducesMoreBlocks - Add gpuMemoryUtilization parameter to CalculateKVBlocks doc comment All tests pass.

- Add inference-sim binary to .gitignore - Replace hardcoded 0.9 with gpuMemoryUtilization variable in cmd/replay.go - Add validation guard for gpuMemoryUtilization in cmd/replay.go - Add logrus.Info logs showing gpu-memory-utilization value in both runCmd and replayCmd - Update flag description to specify range (0, 1.0] - Rename test cases: 'below range' -> 'zero is invalid', 'above range' -> 'above 1.0 is invalid' - Add invariant test TestCalculateKVBlocks_GpuMemoryUtilization_HigherUtilProducesMoreBlocks - Add gpuMemoryUtilization parameter to CalculateKVBlocks doc comment - Revert sim/latency/config.go and config_test.go to match main (TFlopsFP8 changes belong to PR #765) All tests pass.

- Add inference-sim binary to .gitignore - Remove inference-sim binary from git tracking - Replace hardcoded 0.9 with gpuMemoryUtilization variable in cmd/replay.go - Add validation guard for gpuMemoryUtilization in cmd/replay.go - Add logrus.Info logs showing gpu-memory-utilization value in both runCmd and replayCmd - Update flag description to specify range (0, 1.0] - Rename test cases: 'below range' -> 'zero is invalid', 'above range' -> 'above 1.0 is invalid' - Add invariant test TestCalculateKVBlocks_GpuMemoryUtilization_HigherUtilProducesMoreBlocks - Add gpuMemoryUtilization parameter to CalculateKVBlocks doc comment - Revert sim/latency/config.go and config_test.go to match main (TFlopsFP8 changes belong to PR #765) All tests pass.

* feat(sim): add GpuMemoryUtilization field to KVCacheConfig (BC-1, BC-2) - Add GpuMemoryUtilization *float64 field to KVCacheConfig struct - Update NewKVCacheConfig constructor with validation (0.5-1.0 range, NaN/Inf guards) - Update all 92 construction sites to pass nil (preserves current 0.9 default) - Add tests for validation and nil handling Implements BC-1: GpuMemoryUtilization field accepts valid values (0.5-1.0) Implements BC-2: Constructor validates range and rejects invalid values * feat(latency): add gpuMemoryUtilization parameter to CalculateKVBlocks (BC-3) - Add gpuMemoryUtilization parameter to CalculateKVBlocks function - Remove hardcoded gpuMemUtil constant (0.9) - Add validation for range [0.5, 1.0] with NaN/Inf guards - Update all 38 call sites to pass 0.9 (preserves current behavior) - Add test for parameter validation Implements BC-3: CalculateKVBlocks accepts gpuMemoryUtilization parameter * feat(cmd): add --gpu-memory-utilization CLI flag Implements BC-4: CLI flag with validation and threading to config/latency. - Added --gpu-memory-utilization flag (default: 0.9, range: 0.5-1.0) - Added validation with NaN/Inf guards - Threaded to NewKVCacheConfig and CalculateKVBlocks calls - All tests pass * refactor: remove unnecessary GpuMemoryUtilization field from KVCacheConfig The GpuMemoryUtilization field was only used as an input to CalculateKVBlocks() for capacity planning, but was never read by the KV cache runtime. This refactor removes it from KVCacheConfig, keeping it only as a parameter to CalculateKVBlocks(). Changes: - Removed GpuMemoryUtilization field from KVCacheConfig struct - Updated NewKVCacheConfig() constructor to remove the parameter - Updated all call sites (cmd/ and tests) to remove the parameter - CLI flag --gpu-memory-utilization still works correctly This separates calculation inputs from runtime configuration, resulting in a cleaner API and reducing unnecessary parameter passing across 100+ test sites. * Fix gpu_memory_utilization validation to match vLLM range (0, 1.0] The original [0.5, 1.0] range was overly restrictive and incompatible with vLLM's actual validation (gt=0, le=1). This change: - Updates CalculateKVBlocks validation from [0.5, 1.0] to (0, 1.0] - Updates CLI validation in cmd/root.go to match - Updates test cases to verify 0.3 is valid (commonly used in vLLM tests) - Adds tests for boundary cases (0.0, negative values) This makes BLIS compatible with valid vLLM configurations using lower memory utilization values (0.1-0.4) for testing, memory-constrained scenarios, and multi-instance deployments. * Revert whitespace changes in batch_formation_test.go The whitespace alignment changes were unnecessary formatting changes that should not be part of this PR. Reverting to main branch version. * fix: address PR #771 review comments - Add inference-sim binary to .gitignore - Remove inference-sim binary from git tracking - Replace hardcoded 0.9 with gpuMemoryUtilization variable in cmd/replay.go - Add validation guard for gpuMemoryUtilization in cmd/replay.go - Add logrus.Info logs showing gpu-memory-utilization value in both runCmd and replayCmd - Update flag description to specify range (0, 1.0] - Rename test cases: 'below range' -> 'zero is invalid', 'above range' -> 'above 1.0 is invalid' - Add invariant test TestCalculateKVBlocks_GpuMemoryUtilization_HigherUtilProducesMoreBlocks - Add gpuMemoryUtilization parameter to CalculateKVBlocks doc comment - Revert sim/latency/config.go and config_test.go to match main (TFlopsFP8 changes belong to PR #765) All tests pass.

* feat(infra): Phase 1A — nodes, GPUs, and instance lifecycle Add node/GPU placement, instance lifecycle management, multi-model routing, and per-model metrics to the cluster simulator. Co-Authored-By: Claude Opus 4.6 <[email protected]> * chore: add AGENTS.md to .gitignore - AGENTS.md is a generated file for agent context - Should not be committed to the repository * fix(cluster): prevent double-counting in drain redirect policy Fixes INV-1 conservation violation when DrainRedirect policy re-injects queued requests. Previously, redirected requests were counted twice in CompletedRequests: once when initially injected and again when completed after redirection. Changes: - Add Request.Redirected field to track re-injected requests - Mark requests as redirected in drainRedirect.Drain() before re-injection - Skip CompletedRequests increment in recordRequestCompletion() for redirected requests - Add TestInstanceLifecycle_RedirectDrainPreservesConservation to verify fix The fix ensures INV-1 (request conservation) holds: injected = completed + queued + running + dropped + timed_out Addresses PR #697 review feedback on drain redirect conservation. * fix: address PR #697 review comments - Document DrainPolicy as Phase 1C infrastructure - Fix warm-up request overcounting by checking warmUpRemaining - Fix drain callback memory leak in MarkNodeTerminated - Add named constants for lifecycle event priorities - Clarify drainWait GPU release and swap-remove logic - Add .bob/ to .gitignore * Fix warm-up TTFT penalty implementation - Initialize warmUpRemaining for all instances in backward-compat mode - Fix indentation in cluster_event.go - Update warm-up recording to track first N requests - Adjust test expectations to account for queueing effects Fixes build and test failures in PR. * chore: update .gitignore to ignore .bob/notes instead of entire .bob/ directory * feat(latency): decouple quantized weight precision from compute dtype (#698) * feat(latency): decouple quantized weight precision from compute dtype (#443) Roofline and KV capacity calculations now correctly use quantized weight precision (e.g. 0.5 bytes/param for W4A16) for weight memory while keeping the compute dtype (e.g. 2.0 for bfloat16) for KV cache and activations. - Add WeightBytesPerParam field to ModelConfig with zero-value sentinel - Parse quantization_config from HuggingFace config.json (GPTQ, AWQ, FP8) - Add --weight-bytes-per-param CLI flag for manual override - Update calculateMemoryAccessBytes() and computeModelWeightBytes() - Add validation in ValidateRooflineConfig - 18 new tests covering all behavioral contracts Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix(latency): convergence R1 — blackbox path parity, KV validation, quant warnings Address 3 IMPORTANT findings from convergence review Round 1: 1. Apply --weight-bytes-per-param CLI override in blackbox KV auto-calc path (R23: code path parity with analytical backend) 2. Add WeightBytesPerParam validation in CalculateKVBlocks public API 3. Warn when quantization_config is present but weight precision could not be auto-detected (both analytical and blackbox paths) Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix(cmd): convergence R2 — early flag validation, backend-specific logs Move --weight-bytes-per-param validation before backend switch to avoid Fatalf inside the best-effort blackbox block (preserves fall-through contract). Make weight precision log backend-specific: roofline uses it for step time; trained-roofline/crossmodel only for KV capacity. Co-Authored-By: Claude Opus 4.6 <[email protected]> * style: gofmt struct/var alignment after WeightBytesPerParam addition Convergence R3: gofmt -w to re-align ModelConfig struct fields and cmd/root.go var block after longest field name changed. Co-Authored-By: Claude Opus 4.6 <[email protected]> * style: gofmt const/map alignment in kv_capacity files Co-Authored-By: Claude Opus 4.6 <[email protected]> * docs: document MFU calibration approximation for quantized models Add code comment in rooflineStepTime noting that MFU values were calibrated against FP16 measurements. For quantized models (W4A16), reduced weight bandwidth shifts the roofline crossover, producing conservative estimates — safe for capacity planning. Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix(latency): address PR review — FP8 case-insensitive match, test coverage, validation comment - Use strings.EqualFold for FP8 quant_method matching (case-insensitive) - Add test for FP8 with bits=8 present (verifies bits-first path agrees) - Document why WeightBytesPerParam > BytesPerParam is accepted in validation Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat(latency): complete three-tier quantized weight detection, remove CLI flag Add compressed-tensors parsing (config_groups.*.weights.num_bits) to close the gap where w8a8 models silently fell back to torch_dtype. Add model name convention detection (w4a16, fp8) as a second-tier fallback. Remove the --weight-bytes-per-param CLI flag — all three issue #443 options are now covered by auto-detection, making the manual override unnecessary. Detection chain: quantization_config → model name → torch_dtype fallback. Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix(latency): address PR review — FP8 case-insensitive match, test coverage, validation comment C1: Extract applyWeightPrecisionFallback shared helper (cmd/hfconfig.go) and add model name fallback to cmd/replay.go roofline path (was missing). Three call sites (root.go blackbox, root.go roofline, replay.go) now share identical fallback + logging logic. C2: Sort config_groups keys before iteration in compressed-tensors parsing (INV-6 determinism). I2: Add string-to-int coercion for quantization_config.bits field (handles "bits": "4" from some HF configs). I3: Add TestRooflineStepTime_W4A16_LowerThanFP16_MemoryBoundDecode end-to-end test verifying quantized model produces lower decode step time than FP16 in memory-bound regime. I4: Add TestValidateRooflineConfig_InfWeightBytesPerParam_ReturnsError covering +Inf validation gap. Minor: Remove double blank line in root.go init(). Co-Authored-By: Claude Opus 4.6 <[email protected]> * docs: update quantization support language across guides, references, and extension recipes Five documentation pages still claimed quantization was "not yet modeled" and recommended blackbox mode, contradicting the three-tier auto-detection merged in this branch. Addresses PR #698 review comments D1–D5. Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix(cmd): warn when trained-roofline ignores quantized weight precision (I4) Trained-roofline hardcodes FP16 bytesPerElement to match its training pipeline, so WeightBytesPerParam only affects KV capacity, not step time. Previously the CLI logged "quantized model detected" without mentioning this limitation, which was misleading. Now emits an explicit warning in both run and replay paths. Addresses review item I4 from PR #698 convergence review. Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: address cosmetic review comments - Add bitsPerByte constant to replace magic number 8.0 (issue #6) - Improve roofline approximation comment with quantitative guidance (issue #8) * Address PR #698 review feedback: refactor quantization parsing and improve code quality - Extract quantization parsing into parseQuantizationConfig helper (Issue #4) - Consolidate duplicated trained-roofline warning into warnTrainedRooflineQuantization helper (Issue #1) - Add validation warning when WeightBytesPerParam > BytesPerParam (Issue #2) - Add debug logging for malformed compressed-tensors config_groups (Issue #3) - Add error handling for invalid bits string-to-int coercion (Issue #5) - Add comment explaining first-match semantics in compressed-tensors parsing (Issue #7) - Fix inconsistent spacing in string concatenation (Issue #9) All changes maintain backward compatibility and pass existing tests. --------- Co-authored-by: Claude Opus 4.6 <[email protected]> * feat(sim): add L40S GPU and FP8 compute support (#765) * feat(latency): decouple quantized weight precision from compute dtype (#443) Roofline and KV capacity calculations now correctly use quantized weight precision (e.g. 0.5 bytes/param for W4A16) for weight memory while keeping the compute dtype (e.g. 2.0 for bfloat16) for KV cache and activations. - Add WeightBytesPerParam field to ModelConfig with zero-value sentinel - Parse quantization_config from HuggingFace config.json (GPTQ, AWQ, FP8) - Add --weight-bytes-per-param CLI flag for manual override - Update calculateMemoryAccessBytes() and computeModelWeightBytes() - Add validation in ValidateRooflineConfig - 18 new tests covering all behavioral contracts Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix(latency): convergence R1 — blackbox path parity, KV validation, quant warnings Address 3 IMPORTANT findings from convergence review Round 1: 1. Apply --weight-bytes-per-param CLI override in blackbox KV auto-calc path (R23: code path parity with analytical backend) 2. Add WeightBytesPerParam validation in CalculateKVBlocks public API 3. Warn when quantization_config is present but weight precision could not be auto-detected (both analytical and blackbox paths) Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix(cmd): convergence R2 — early flag validation, backend-specific logs Move --weight-bytes-per-param validation before backend switch to avoid Fatalf inside the best-effort blackbox block (preserves fall-through contract). Make weight precision log backend-specific: roofline uses it for step time; trained-roofline/crossmodel only for KV capacity. Co-Authored-By: Claude Opus 4.6 <[email protected]> * style: gofmt struct/var alignment after WeightBytesPerParam addition Convergence R3: gofmt -w to re-align ModelConfig struct fields and cmd/root.go var block after longest field name changed. Co-Authored-By: Claude Opus 4.6 <[email protected]> * style: gofmt const/map alignment in kv_capacity files Co-Authored-By: Claude Opus 4.6 <[email protected]> * docs: document MFU calibration approximation for quantized models Add code comment in rooflineStepTime noting that MFU values were calibrated against FP16 measurements. For quantized models (W4A16), reduced weight bandwidth shifts the roofline crossover, producing conservative estimates — safe for capacity planning. Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix(latency): address PR review — FP8 case-insensitive match, test coverage, validation comment - Use strings.EqualFold for FP8 quant_method matching (case-insensitive) - Add test for FP8 with bits=8 present (verifies bits-first path agrees) - Document why WeightBytesPerParam > BytesPerParam is accepted in validation Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat(latency): complete three-tier quantized weight detection, remove CLI flag Add compressed-tensors parsing (config_groups.*.weights.num_bits) to close the gap where w8a8 models silently fell back to torch_dtype. Add model name convention detection (w4a16, fp8) as a second-tier fallback. Remove the --weight-bytes-per-param CLI flag — all three issue #443 options are now covered by auto-detection, making the manual override unnecessary. Detection chain: quantization_config → model name → torch_dtype fallback. Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix(latency): address PR review — FP8 case-insensitive match, test coverage, validation comment C1: Extract applyWeightPrecisionFallback shared helper (cmd/hfconfig.go) and add model name fallback to cmd/replay.go roofline path (was missing). Three call sites (root.go blackbox, root.go roofline, replay.go) now share identical fallback + logging logic. C2: Sort config_groups keys before iteration in compressed-tensors parsing (INV-6 determinism). I2: Add string-to-int coercion for quantization_config.bits field (handles "bits": "4" from some HF configs). I3: Add TestRooflineStepTime_W4A16_LowerThanFP16_MemoryBoundDecode end-to-end test verifying quantized model produces lower decode step time than FP16 in memory-bound regime. I4: Add TestValidateRooflineConfig_InfWeightBytesPerParam_ReturnsError covering +Inf validation gap. Minor: Remove double blank line in root.go init(). Co-Authored-By: Claude Opus 4.6 <[email protected]> * docs: update quantization support language across guides, references, and extension recipes Five documentation pages still claimed quantization was "not yet modeled" and recommended blackbox mode, contradicting the three-tier auto-detection merged in this branch. Addresses PR #698 review comments D1–D5. Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix(cmd): warn when trained-roofline ignores quantized weight precision (I4) Trained-roofline hardcodes FP16 bytesPerElement to match its training pipeline, so WeightBytesPerParam only affects KV capacity, not step time. Previously the CLI logged "quantized model detected" without mentioning this limitation, which was misleading. Now emits an explicit warning in both run and replay paths. Addresses review item I4 from PR #698 convergence review. Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: address cosmetic review comments - Add bitsPerByte constant to replace magic number 8.0 (issue #6) - Improve roofline approximation comment with quantitative guidance (issue #8) * Address PR #698 review feedback: refactor quantization parsing and improve code quality - Extract quantization parsing into parseQuantizationConfig helper (Issue #4) - Consolidate duplicated trained-roofline warning into warnTrainedRooflineQuantization helper (Issue #1) - Add validation warning when WeightBytesPerParam > BytesPerParam (Issue #2) - Add debug logging for malformed compressed-tensors config_groups (Issue #3) - Add error handling for invalid bits string-to-int coercion (Issue #5) - Add comment explaining first-match semantics in compressed-tensors parsing (Issue #7) - Fix inconsistent spacing in string concatenation (Issue #9) All changes maintain backward compatibility and pass existing tests. * feat(sim): add L40S GPU and FP8 compute support - Add L40S hardware configuration (362.05 TFLOPS, 48GB, 0.864 TB/s) - Add TFlopsFP8 field to HardwareCalib for native FP8 tensor core support - Update H100 with TFlopsFP8=1979.0 (2× FP16 rate) and adjusted MFU values - Update A100-SXM and A100-80 with TFlopsFP8=0 (no native FP8 support) - Implement FP8 compute selection in roofline model based on weight precision - Add comprehensive tests for FP8 compute selection logic Fixes #762 Co-Authored-By: Claude <[email protected]> * docs: add MFU justification and validation tests - Add inline documentation to hardware_config.json linking to Discussion #589 - Add comprehensive MFU validation tests in sim/config_test.go - Validates MFU ranges (0 < MFU < 1) - Validates MfuDecode < MfuPrefill relationship - Tests all GPU types (H100, A100, L40S) - Update docs/reference/models.md with MFU calibration info box - All tests pass Addresses review findings from quick-review * feat(latency): add TFlopsFP8 validation (R3) Add validation for HardwareCalib.TFlopsFP8 in ValidateRooflineConfig, following the same optional-field pattern as MemoryGiB: check if non-zero, then reject NaN/Inf/negative values. Includes test coverage for NaN and negative TFlopsFP8 cases. * fix: address review nits - roofline.go:217: Fix comment wording from 'upcasted' to 'dequantized to FP16 during GEMM' - hardware_config.json:8: Restore H100 mfuDecode to 0.30 for consistency with other entries * docs(roofline): clarify FP8 compute rate selection logic Improve comment to explicitly document that == 1.0 is intentional: - FP8 models (exactly 1.0 byte/param) use FP8 compute rate on H100 - Sub-FP8 formats (e.g., W4A16 at 0.5 bytes/param) dequantize to FP16 during GEMM This addresses the review comment about == 1.0 exact equality. The behavior is correct: only true FP8 models use FP8 tensor cores. W4A16 and other sub-FP8 formats use FP16 compute after dequantization, as validated by TestRooflineStepTime_FP8ComputeSelection_EdgeCases. --------- Co-authored-by: Claude Opus 4.6 <[email protected]> * fix(cluster): address sriumcp review — conservation, replay anomaly, non-variadic CollectRawMetrics - sim/simulator.go: Remove incorrect `if !req.Redirected` guard on CompletedRequests++. The guard caused redirected requests to vanish from INV-1 accounting: source's InjectedRequests=0 (drained from WaitQ before completion), destination's InjectedRequests=0 (skipped CompletedRequests). Destination is the sole completion site so incrementing there preserves conservation. - cmd/replay.go: Add `|| rawMetrics.RoutingRejections > 0` to anomaly condition. Clusters where all failures are routing rejections (no routable instances) silently omitted the anomaly summary block (I3 from sriumcp review). - sim/cluster/metrics.go: Make CollectRawMetrics routingRejections parameter explicit (non-variadic). Prevents call sites from silently passing 0 and missing routing rejections. Updated all test call sites to pass 0 explicitly. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * revert(ci): remove skip-cache from golangci-lint step Unrelated to Phase 1A changes. skip-cache: true was added during development but should not be merged to main. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix(cluster): address sriumcp Round 2 — test, stale comment, duplicate Metrics.Requests - sim/cluster/instance_lifecycle_test.go: Rewrite conservation test to actually seed requests into inst0's WaitQ before drain. Previous version used a manual event loop on an empty queue — DrainWaitQueue() returned [] and no redirection ever occurred. New version uses inst0.sim.EnqueueRequest directly + empty workload so Run() doesn't push duplicate arrivals. Also adds pre/post assertions: QueueDepth==0, inFlightRequests==0, and clusterEvents non-empty after drain. - sim/request.go: Update Redirected field comment to reflect actual behavior. Previous comment said "completion accounting is skipped" — opposite of what simulator.go:recordRequestCompletion now does. - sim/cluster/infra_lifecycle_event.go: Delete stale Metrics.Requests entry for redirected requests before re-injection. Source registered the request at EnqueueRequest time; DrainWaitQueue empties WaitQ but left the entry. Destination re-registers on re-enqueue, causing a spurious "duplicate request ID" WARN in aggregateMetrics() for every redirected request. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * docs: address sriumcp documentation gaps (GAP 1-4) GAP 1 — configuration.md: Add node_pools and instance_lifecycle YAML schema to the Policy Bundle section so users can discover and configure Phase 1A features. Both are YAML-only (no CLI flags). Add a note block explaining backward compatibility. Update DeploymentConfig row in the config-to-flag mapping table to note the YAML-only fields. GAP 2 — results.md Anomaly Counters: Rename "Rejected Requests" to "Rejected Requests (Admission)" to match actual CLI output label. Add new "Rejected Requests (Routing)" row explaining when it fires (no routable instances — all Loading/Draining) and the remediation action. GAP 3 — results.md Per-Model Metrics: Change mean= to p50= in the example output block to match printPerModelMetrics which uses m.TTFT.P50. Add tok/s to the Throughput example line to match actual output format. GAP 4 — results.md per_model JSON: Add table documenting the per_model key in --results-path JSON output (omitted when no model tags present), with field-by-field description. Co-Authored-By: Claude Sonnet 4.6 <[email protected]> --------- Co-authored-by: tantawi <[email protected]> Co-authored-by: Claude Opus 4.6 <[email protected]> Co-authored-by: Dipanwita Guhathakurta <[email protected]> Co-authored-by: Srinivasan Parthasarathy <[email protected]>

susiejojo and others added 14 commits March 19, 2026 10:14

style: gofmt struct/var alignment after WeightBytesPerParam addition

eebb2f1

Convergence R3: gofmt -w to re-align ModelConfig struct fields and cmd/root.go var block after longest field name changed. Co-Authored-By: Claude Opus 4.6 <[email protected]>

style: gofmt const/map alignment in kv_capacity files

6b9ae7c

Co-Authored-By: Claude Opus 4.6 <[email protected]>

fix: address cosmetic review comments

5dcc5cd

- Add bitsPerByte constant to replace magic number 8.0 (issue #6) - Improve roofline approximation comment with quantitative guidance (issue #8)

susiejojo force-pushed the pr762-fp8-hardware-support branch 2 times, most recently from fbec9c4 to 66205db Compare March 19, 2026 19:16

susiejojo requested a review from sriumcp March 19, 2026 19:19

This comment was marked as duplicate.

Sign in to view

sriumcp reviewed Mar 19, 2026

View reviewed changes

susiejojo force-pushed the pr762-fp8-hardware-support branch from 2bc8682 to 06c5d4d Compare March 19, 2026 20:53

susiejojo force-pushed the pr762-fp8-hardware-support branch from 06c5d4d to 6a041e0 Compare March 19, 2026 20:56

susiejojo force-pushed the pr762-fp8-hardware-support branch from 6a041e0 to a1bde43 Compare March 19, 2026 20:58

susiejojo force-pushed the pr762-fp8-hardware-support branch from a1bde43 to be177ab Compare March 20, 2026 14:24

Merge main into pr762-fp8-hardware-support

35c04b6

sriumcp reviewed Mar 20, 2026

View reviewed changes

sim/model_hardware_config.go Show resolved Hide resolved

sriumcp reviewed Mar 20, 2026

View reviewed changes

sim/latency/roofline.go Show resolved Hide resolved

sriumcp reviewed Mar 20, 2026

View reviewed changes

sim/latency/roofline.go Outdated Show resolved Hide resolved

sriumcp reviewed Mar 20, 2026

View reviewed changes

hardware_config.json Outdated Show resolved Hide resolved

susiejojo added 3 commits March 20, 2026 10:47

fix: address review nits

45d4cd5

- roofline.go:217: Fix comment wording from 'upcasted' to 'dequantized to FP16 during GEMM' - hardware_config.json:8: Restore H100 mfuDecode to 0.30 for consistency with other entries

susiejojo requested a review from sriumcp March 20, 2026 15:10

sriumcp approved these changes Mar 20, 2026

View reviewed changes

sriumcp merged commit cad4191 into main Mar 20, 2026
4 of 5 checks passed

sriumcp mentioned this pull request Mar 20, 2026

feat: add --gpu-memory-utilization CLI flag #771

Merged

susiejojo deleted the pr762-fp8-hardware-support branch March 20, 2026 18:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(sim): add L40S GPU and FP8 compute support#765

feat(sim): add L40S GPU and FP8 compute support#765
sriumcp merged 19 commits intomainfrom
pr762-fp8-hardware-support

susiejojo commented Mar 19, 2026 •

edited

Loading

Uh oh!

This comment was marked as duplicate.

Uh oh!

sriumcp left a comment

Uh oh!

sriumcp left a comment

Uh oh!

sriumcp Mar 20, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

susiejojo commented Mar 20, 2026 •

edited

Loading

Uh oh!

sriumcp left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

susiejojo commented Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

1. L40S GPU Support

2. FP8 Compute Support

3. Roofline Model Updates

4. MFU Calibration Updates

MFU Value Justification

Testing

Uh oh!

This comment was marked as duplicate.

Uh oh!

sriumcp left a comment

Choose a reason for hiding this comment

Multi-Perspective Code Review

🔴 High Priority

🟡 Medium Priority

🟢 Low Priority

✅ Positive Findings

Uh oh!

sriumcp left a comment

Choose a reason for hiding this comment

Follow-up review — unaddressed items from previous round

🔴 R3 violation — TFlopsFP8 not validated in ValidateRooflineConfig

🟡 R4 concern — two HardwareCalib construction sites omit TFlopsFP8

🟢 Minor nits

Uh oh!

sriumcp Mar 20, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

susiejojo commented Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Verification: W4A16 uses FP16 compute rate (not FP8)

Key Findings from vLLM:

Conclusion

Uh oh!

sriumcp left a comment

Choose a reason for hiding this comment

LGTM — ready to merge with one micro-nit still open

Addressed since previous round

Still open (non-blocking)

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

susiejojo commented Mar 19, 2026 •

edited

Loading

🔴 R3 violation — `TFlopsFP8` not validated in `ValidateRooflineConfig`

🟡 R4 concern — two `HardwareCalib` construction sites omit `TFlopsFP8`

susiejojo commented Mar 20, 2026 •

edited

Loading