[
    {
        "role": "user",
        "content": "when%20running%20Nemotron-3-Nano%20on%20nvidia%20spark%2C%20what%20is%20tokens%20per%20second%3F"
    },
    {
        "role": "assistant",
        "content": "The tokens-per-second (t/s) rate you can expect when running **Nemotron-3-Nano** on **NVIDIA DGX Spark** depends heavily on your setup: model mode, context length, quantization, inference engine, concurrency, and whether it's text-only or multimodal input. Below are the key metrics based on recent benchmarks and official reports, along with estimates for DGX Spark.\n\n---\n\n## \ud83d\udd0d Official Benchmarks & Reported Throughput\n\n1. **Text Generation on DGX B200 (FP8)**  \n   - With TensorRT-LLM + AutoDeploy:  \n     * Up to **350 tokens/sec per user** in interactive scenarios ([developer.nvidia.com](https://developer.nvidia.com/blog/automating-inference-optimizations-with-nvidia-tensorrt-llm-autodeploy/?utm_source=openai))  \n     * Up to **~13,000 tokens/sec** in high-throughput (batched) settings ([developer.nvidia.com](https://developer.nvidia.com/blog/automating-inference-optimizations-with-nvidia-tensorrt-llm-autodeploy/?utm_source=openai))  \n\n2. **Nemotron 3 Nano Omni on B200 (NVFP4 quantization)**  \n   - Single-stream (concurrency = 1): **> 500 output tokens/sec** for text-only generation ([research.nvidia.com](https://research.nvidia.com/labs/nemotron/files/NVIDIA-Nemotron-3-Omni-report.pdf?utm_source=openai))  \n   - Multidocument workloads & longer multimodal inputs: up to **5,000 output tokens/sec** at max concurrency ([research.nvidia.com](https://research.nvidia.com/labs/nemotron/files/NVIDIA-Nemotron-3-Omni-report.pdf?utm_source=openai))  \n\n3. **Speed vs. Accuracy Ranking (Artificial Analysis)**  \n   - The \u201cReasoning\u201d version of Nemotron-3-Nano is reported around **200 output tokens/sec** on typical API/provider settings in benchmarked environments. ([artificialanalysis.ai](https://artificialanalysis.ai/models/nvidia-nemotron-3-nano-30b-a3b-reasoning/?utm_source=openai))  \n\n4. **DGX Spark / Spark Arena Forum Report**  \n   - When running Nemotron-3-Nano-Omni-30B-A3B-Reasoning in NVFP4 under vLLM, one user reported ~**56.96 tokens/sec** on text generation using DGX Spark. ([forums.developer.nvidia.com](https://forums.developer.nvidia.com/t/benchmark-nvidia-nemotron-3-nano-omni-30b-a3b-reasoning-nvfp4/368566?utm_source=openai))  \n\n---\n\n## \u2699\ufe0f Expected Tokens/sec Performance on DGX Spark\n\nGiven the above, here\u2019s what to reasonably expect if you run Nemotron-3-Nano on **DGX Spark**:\n\n| Scenario | Likely Setup | Expected Output Tokens/sec |\n|----------|---------------|-----------------------------|\n| **Text-only, moderate context (~8\u201316K tokens), single-user** | FP8 or NVFP4 quantization, using TensorRT-LLM or vLLM | **\u2248 150-400 t/s** (possibly more with optimal setup) |\n| **Text with longer input/multidocument, some concurrency** | Same quantization, more context effects play in | **\u2248 50-150 t/s per stream**, aggregate higher with multiple streams |\n| **Full multimodal or \u201cOmni\u201d mode with images/videos** | Using Omni-specific setups (e.g. Nano Omni) | Could achieve **several hundred t/s** but more likely lower per stream due to heavier compute ([research.nvidia.com](https://research.nvidia.com/labs/nemotron/files/NVIDIA-Nemotron-3-Omni-report.pdf?utm_source=openai)) |\n\n---\n\n## \u26a0\ufe0f What Low Tokens/sec (like ~56 t/s) Indicates\n\n- The **~56.96 t/s** figure with DGX Spark + vLLM reflects a setup with reasoning mode, NVFP4 quantization, likely with longer context or other constraints (e.g. smaller batch size or higher latency targets). ([forums.developer.nvidia.com](https://forums.developer.nvidia.com/t/benchmark-nvidia-nemotron-3-nano-omni-30b-a3b-reasoning-nvfp4/368566?utm_source=openai))  \n- It\u2019s substantially lower than peak benchmarks because DGX Spark has many GPUs (e.g., GB10 unit) and is optimized for distributed workflows. Single-stream performance tends to be lower unless you're fully utilizing the hardware.\n\n---\n\n## \u2705 Bottom Line Estimate\n\nIf you\u2019re running **Nemotron-3-Nano** **on DGX Spark**, reasonable expectations are:\n\n- **Best case** (well-tuned, FP8/NVFP4, short to moderate context, ideal engine): **200\u2013400 tokens/sec per stream** for text generation.  \n- **Typical case** with some constraints (longer context, concurrency, reasoning mode): **50\u2013150 t/s**.\n\nIf you tell me your exact configuration (quantization, context length, inference engine, concurrency), I can give a more precise estimate. Do you want me to compute based on your setup?"
    }
]