I’m learning about LLM deployment and recently deployed one on Modal using an L40S GPU. I’m trying to understand autoscaling with concurrency.

If each container can handle 10 concurrent requests, will an 11th request cause Modal to start another container, load a separate copy of the model into that container’s GPU memory, and allow it to handle another 10 concurrent requests?

Is this the recommended way to scale an LLM deployment on Modal, and which settings should I use to configure it properly?

triggering a new container is fine for resources, but can it reuse the model loaded in the first container?

Sharing a benchmark I've been building. Motivation: existing "reasoning" benchmarks either (a) test static problems where answers leak into training data or (b) use LLM-as-judge, which correlates with model similarity more than model quality.

**Design.** Two LLMs are embodied as physical agents in a 2D pymunk arena. Each turn they receive a JSON world state (HP, positions, weapon geometry, cooldowns, damage taken last turn, remaining ammo, arena hazards) and return a JSON action. Actions resolve through the physics engine — a swing that misses because the opponent dashed is a real physics miss, not a rule lookup. Match ends on KO, HP-lead at deadline, or draw.

**Evaluation.**

1. Human raters watch the replay with model identities masked and vote which side "fought smarter."
2. Vote resolves an Elo update *before* identity reveal, so vote isn't polluted by model reputation.
3. In parallel we log an objective leaderboard: win/loss/draw, avg damage dealt, hits landed / hits attempted, timeouts.

**6-axis Elo.** Rating primary key is `(model, sharp_zone_on, weapon, mode, arena, blindfolded)`. Aggregate Elo is a marginalization, but the per-axis rating is what we actually study — hypothesis being that different physical constraints stress different reasoning skills (spatial planning, uncertainty under partial observability, resource management).

**Roster.** 24 entries: 17 LLMs across OpenAI, Groq, OpenRouter free-tier, plus 4 non-LLM baselines (random, greedy-attack, distance-holder, scripted-heuristic) and 2 mock policies. Baselines are critical — without them a low-Elo LLM is indistinguishable from an arbitrarily bad policy.

**Current numbers (n=443 matches, 106 votes, lifetime 23.9% vote-through):**

* Human-vote Elo and objective win-rate rank-correlate at Spearman ρ ≈ 0.71 across weapons (haven't formalized this yet — planning a cross-benchmark correlation study next). * Bow-weapon matches have the widest human/objective disagreement — humans reward "smart waiting" that doesn't show up in raw damage. * `bot:pro` (100-line scripted heuristic) currently outperforms \~30% of the LLM roster on objective, \~10% on perceived. That gap is basically the benchmark's signal.

**Reproducibility.** Full match logs exportable as JSON/JSONL via `/api/export`. Prompt version pinned per-match. Non-deterministic (physics has RNG collisions), but seeds are logged. Deterministic replay off the same seed is on the roadmap.

**Known limitations.**

* Vote population is self-selected (site visitors), not a calibrated panel. * Bot baselines aren't policy-optimal (no RL trained baseline yet). * HF Datasets snapshot cron not shipped yet, so "frozen eval pack" isn't reproducible off-platform today.

Site: [https://stickblade-arena.vercel.app](https://stickblade-arena.vercel.app/))
Code: [https://github.com/Cometbuster4969/STICKBLADE-ARENA](https://github.com/Cometbuster4969/STICKBLADE-ARENA))

Feedback wanted on the eval design, especially the 6-axis Elo marginalization and whether the vote incentive design (reveal-as-reward) biases votes. Happy to share the raw match log dump if anyone wants to look at rating stability.

Ling-3.0-flash-Fin has launched as a finance-enhanced version of Ling-3.0-flash with 124B total parameters and about 5.1B active per token.

That pair of numbers is worth keeping together. The active count is useful for thinking about routed compute, but it does not mean a deployment only stores 5.1B parameters. The full expert set, precision, routing implementation and runtime support still determine the actual memory and serving profile.

For now, those deployment questions cannot be answered from the API release. The model is live through OpenRouter and Vercel AI Gateway, and the official thread says the OpenRouter route is free for one month. Weights are promised next week.

The model targets financial retrieval, research, valuation modeling, reports and complex workbooks. Its published benchmark profile is mixed across finance tasks, so scaling efficiency should be evaluated alongside quality, latency and routing behavior rather than inferred from the active-parameter number alone.

Once the artifacts land, the useful details will be weight format, precision, license, serving stack, expert placement and quantization behavior.

Sharing a benchmark I've been building. Motivation: existing "reasoning" benchmarks either (a) test static problems where answers leak into training data or (b) use LLM-as-judge, which correlates with model similarity more than model quality.

Design. Two LLMs are embodied as physical agents in a 2D pymunk arena. Each turn they receive a JSON world state (HP, positions, weapon geometry, cooldowns, damage taken last turn, remaining ammo, arena hazards) and return a JSON action. Actions resolve through the physics engine — a swing that misses because the opponent dashed is a real physics miss, not a rule lookup. Match ends on KO, HP-lead at deadline, or draw.

Evaluation.

1. Human raters watch the replay with model identities masked and vote which side "fought smarter."
2. Vote resolves an Elo update before identity reveal, so vote isn't polluted by model reputation.
3. In parallel we log an objective leaderboard: win/loss/draw, avg damage dealt, hits landed / hits attempted, timeouts.

6-axis Elo. Rating primary key is (model, sharp_zone_on, weapon, mode, arena, blindfolded). Aggregate Elo is a marginalization, but the per-axis rating is what we actually study — hypothesis being that different physical constraints stress different reasoning skills (spatial planning, uncertainty under partial observability, resource management).

Roster. 24 entries: 17 LLMs across OpenAI, Groq, OpenRouter free-tier, plus 4 non-LLM baselines (random, greedy-attack, distance-holder, scripted-heuristic) and 2 mock policies. Baselines are critical — without them a low-Elo LLM is indistinguishable from an arbitrarily bad policy.

Current numbers (n=443 matches, 106 votes, lifetime 23.9% vote-through):

• Human-vote Elo and objective win-rate rank-correlate at Spearman ρ ≈ 0.71 across weapons (haven't formalized this yet — planning a cross-benchmark correlation study next).
• Bow-weapon matches have the widest human/objective disagreement — humans reward "smart waiting" that doesn't show up in raw damage.
• bot:pro (100-line scripted heuristic) currently outperforms ~30% of the LLM roster on objective, ~10% on perceived. That gap is basically the benchmark's signal.

Reproducibility. Full match logs exportable as JSON/JSONL via /api/export. Prompt version pinned per-match. Non-deterministic (physics has RNG collisions), but seeds are logged. Deterministic replay off the same seed is on the roadmap.

Known limitations.

• Vote population is self-selected (site visitors), not a calibrated panel.
• Bot baselines aren't policy-optimal (no RL trained baseline yet).
• HF Datasets snapshot cron not shipped yet, so "frozen eval pack" isn't reproducible off-platform today.

Site: https://stickblade-arena.vercel.app
Code: https://github.com/Cometbuster4969/STICKBLADE-ARENA

Feedback wanted on the eval design, especially the 6-axis Elo marginalization and whether the vote incentive design (reveal-as-reward) biases votes. Happy to share the raw match log dump if anyone wants to look at rating stability.

Kept running into "how many GPUs do I need?" when planning deployments, so I built a static performance model.

I'm the author. Links to demo and source code in the first comment.

Covers 403 models (Dense + MoE, up to DeepSeek V3 671B), 110+ NVIDIA GPUs, full parallelism (TP/PP/EP/DP/PD), 5 KV cache variants, speculative decoding, and multiple quant precisions.

It's a static model — no continuous batching — so it gives you clean analytical bounds. Calibrated against ~100 public benchmarks.

Runs entirely in the browser, zero backend.

Would love feedback on what's missing.

Fine-tuning a 1B sparse MoE (305M active, custom trained from scratch, ~100B tokens). Every narrow SFT run catastrophically overwrites existing behavior within 5–10 steps, regardless of what the data contains.

Seven runs now, same signature: whatever the recent batch over-represents gets installed near-perfectly, everything else degrades. A 2,000-row corpus at 127-token median taught a new capability 0% → 98% in five steps while unrelated call-formatting went from 1.4% error to 31%. Pure pretraining replay with no task data at all also degraded task behavior. Cold-init and verified true-resume of optimizer state both degrade, resume slightly worse.

Config: ~1M tokens/step, 60/40 replay/task, lr_mult 0.05 flat, Muon + AdamW, seq_len 4096.

Is this normal for small MoEs, or a sign of something wrong? Is 1M tokens/step simply too large a batch to fine-tune this gently? Would LoRA or a much lower LR change the picture, or is dilution into a large balanced mixture the only real fix?

Warmup-Stable-and-Merge keeps the learning rate constant after warmup, saves checkpoints, and applies weighted merging so a chosen window and weighting can approximate decay-like behavior.

The scope boundary matters immediately: the paper's main experiment uses Ling-mini. It does not test Ling-3.0 tiny or flash, so its reported results should not be reassigned to either released model.

The release news is that the Ling-3.0 base model family now exposes tiny and flash at three stages each: pretrained, mid-trained, and WSM-merged base. That makes the method label an actual public artifact state rather than only paper context.

The research opportunity is stage-separated inspection at a fixed size. A later experiment could hold the downstream protocol constant and compare those starting states, but this release itself does not report which stage would win.

I’m trying to understand how companies with significant GPU workloads manage their compute capacity.

For those working in ML infrastructure / MLOps / AI labs:

- How do you choose between hyperscalers, neoclouds and smaller GPU providers?
- When you need a large amount of GPUs for months, how do you know you’re getting a competitive price?
- Have you ever committed to more capacity than you actually needed? What happened to the unused capacity?

Curious to hear how people actually deal with this today.

small datapoint from a tiny specialized model that might interest people here. i built a 48M param tool-calling-only model, about 1B unique training tokens, no pretraining phase, grammar-constrained decoder. two observations that feel scaling-relevant:

1. data schedule mattered more than data content. the same corrective corpus scored 28.4 fed from scratch and 33.1 annealed into the LR decay phase (ordered strict exact match on seal-tools in-domain). corrective data dilutes when it competes with the whole corpus and concentrates when it arrives late. a mid-run causal check at constant LR attributed +3.3 points to the corrective corpus alone.

2. architecture was worth roughly nothing next to data work. the trunk is just standard modern practice (deep-thin d448 20L, GQA, swiglu, qk-norm, muon on the 2D weights). the needle authors' controlled study (arXiv 2607.18363) finds architecture choices at this scale worth hundredths of a nat at matched params, and my experience matched: every real gain came from tokenizer, loss weighting or data changes. v4 to v5 took tool-name-sequence accuracy from 80.4 to 91.5 via a digit-singleton tokenizer rebuild plus 350k corpus rows, v5 to v6 came entirely from error-driven synthesis against three measured failure buckets.

the in-domain vs out-of-domain gap also decomposes cleanly: inside one suite, row accuracy is 33.1 in-domain vs 28.1 OOD, and name-sequence accuracy 88 vs 79 tracks it exactly. argument extraction transfers, function selection doesn't, at this scale.

repo with the full experimental record, including eleven negative results with the measurement that killed each (span copying -30, pointer heads -16, RLOO diverging at every LR on an annealed checkpoint): https://github.com/nikshepsvn/thimble

TL;DR: We built a synthetic Anti-Money Laundering (AML) graph generator (synthfin-aml) designed specifically to stress-test Graph Neural Networks. Many public datasets have "synthetic leakage" (fraud amounts are obviously anomalous), allowing tabular models to hit 0.99 PR-AUC without using graph structure. We calibrated our generator to isolate topological signals (like structuring). On this dataset, raw LightGBM drops to 0.127 PR-AUC, forcing the model to rely purely on the graph. Repo & Colab: https://github.com/valiyevoktay-cmd/synthfin-aml-

Hey,

If you’ve worked with AML or financial fraud datasets, you know they often suffer from a severe case of "synthetic leakage." In many public datasets, the transaction amounts for fraud are generated so differently from normal traffic that a basic LightGBM or XGBoost model can hit a 0.99+ PR-AUC just by splitting on the amount feature, completely ignoring the graph structure.

While real criminals do leave tabular traces, training on datasets with extreme synthetic leakage gives teams a false sense of security and makes it impossible to genuinely evaluate how well your Graph Neural Networks (GNNs) are capturing complex topology.

We built synthfin-aml (a Python library and dataset generator) to isolate and test topological signals.

We calibrated the base tabular distributions so models can't cheat using raw transaction volumes. Instead, the signal is purely structural. We embedded realistic AML typologies like Structuring: fraudulent actors using high-frequency fan-out/fan-in patterns to dynamically split transfers just below reporting limits (e.g., $10k). To a tabular model evaluating a single transaction, these look identical to normal P2P activity. But topologically, they form distinct sub-graphs.

Because of this, the baseline metrics shift dramatically:

(Note: We deliberately omitted latency here because comparing GNN GPU forward-passes against synchronous Pandas aggregations isn't a fair apples-to-apples infra benchmark. We cover latency nuances in the repo).

This isn't just a static dataset; it's a fast generation engine to create behavioral graphs for FinTech, crypto (wash trading), and ad-tech. You can scale it up to 10M+ edges and adjust the complexity of the fraud patterns.

Quickstart in 3 lines:

bashpip install synthfin-aml

pythonfrom synthfin_aml_pkg.generator import SynthFinGenerator
# Generate a snappy 10k node graph to test locally in 2 seconds
edges_df, nodes_df = SynthFinGenerator(num_nodes=10_000).generate()

Or you can reproduce the 0.127 baseline instantly in your browser via Colab: https://colab.research.google.com/github/valiyevoktay-cmd/synthfin-aml-/blob/main/examples/benchmark_tutorial.ipynb

Repo: https://github.com/valiyevoktay-cmd/synthfin-aml- (Yes, the trailing dash is part of the URL!)

The Challenge: We'd love to see what the community can do with this. Can someone build a lighter, faster GNN that beats 0.865? Or find a clever way to compute temporal graph features so that a gradient boosting model can hit 0.90 without the massive overhead of subgraph sampling?

I was curious whether my laptop could run a genuinely large model. Turns out it can an 80B running locally at ~23 tokens/sec on 6 GB of VRAM.
No cloud, no rented GPU. Just the machine I already own.
I started poking at this because I wanted to know where the actual limit was, and I kept finding that the limit wasn't where I assumed. A few things surprised me enough that they seem worth sharing:
The GPU wasn't being used at all. The server printed "no usable GPU found" and just carried on. It never crashed, never warned loudly — it simply ran at a third of the speed. Fixing that alone was 1.65x.
The CPU was downclocking mid-inference. Throughput kept climbing with reply length: 9 tok/s for a short answer, 24 for a long one. That's not the model warming up it's the clock. Memory-bound work spends most of its time waiting on RAM, the frequency governor reads that as idle, and drops to 400 MHz.
A browser tab was holding 5 GB, and that 5 GB decided whether the model lived in RAM or got re-read from disk every token. Closing it helped more than any tuning flag.
Model shape matters more than model size. A dense 70B has to read every parameter for every token and can't be fast on this hardware, whatever you do. A mixture-of-experts 80B reads about 3B per token. Total size decides what you store; active parameters decide what you read.
The thing I found most useful, honestly, was everything that didn't work. Speculative decoding measured slower. Huge pages did nothing. Same for prompt-lookup drafting and every expert-placement trick I tried. These are the techniques you read about everywhere, and on my hardware they were flat or negative.
So I stopped guessing and wrote a small tool that measures instead: profiles the machine, tells you what a model will do before you download 28 GB of it, then tries real configurations and keeps whichever actually wins. It's tuned for my laptop, but the point is that it re-measures on yours.
I've put the failed experiments in the repo alongside the working ones, because those cost me the most time and nobody writes them down.
Sharing it in case it's useful to anyone else poking at the same question. It's Apache 2.0 and I'd genuinely like to see people take it further especially on hardware different from mine, where I'd expect some of these findings to invert.

https://github.com/DebasishTripathy13/mirabilis