1 · Throughput flatlining
NIC monitoring shows throughput pinned around 9-9.5 Gb/s on a 10G NIC during backup, replication or file transfer. The NIC is saturated, and under saturation applications start seeing latency.
What we do
Hardware repair Server maintenance Maintenance contracts Data recovery Upgrades & expansionBy manufacturer
Dell PowerEdge HPE ProLiant Lenovo ThinkSystem Fujitsu Primergy Supermicro IBM System x / Power Acer Altos
Upgrade · networking
NIC upgrades: 10G is no longer enough for many modern workloads.
Enterprise servers built 5-7 years ago usually ship with onboard 1G/10G NICs. For modern workloads — vSAN, Storage Spaces Direct, Ceph, high-density backup, DR replication — 25G/100G has become the standard. A NIC upgrade is one of the highest-impact interventions available: a few hundred euros of adapter, a real benefit to application performance.
When to upgrade the NIC
Four signs the network is your bottleneck.
2 · Backup window over budget
A nightly backup of 4-6 TB over a 10G NIC typically takes 2-3 hours. Once the window no longer fits (10+ TB), moving to 25G cuts it by 2.5×, 100G by 10×.
3 · Underperforming vSAN / SDS
vSAN works over 10G, but write latency is marginal. 25G+ with RDMA brings vSAN write latency down to levels acceptable even for database VMs. The same applies to Storage Spaces Direct.
4 · Slow VM migration
vMotion / Live Migration of a VM with 100+ GB of RAM takes minutes over 10G, seconds over 25G. On workloads with dynamic HA that is an operational factor, not just a performance one.
Technical constraints
Five things to check.
1 · Available PCIe slots and generation
A 100G NIC needs PCIe Gen4 x16 (or Gen3 x16, with limits); a 25G NIC needs Gen3 x8 as a minimum. Slot availability on enterprise servers has to be verified up front — on smaller systems (1U) NICs compete with HBAs, GPUs and controllers.
2 · Cabling and optics
10G uses SFP+ (DAC up to 7 m, LR optics up to 10 km). 25G uses SFP28 (3 m DAC, LR optics). 100G uses QSFP28 (3 m DAC, various optics). Vendor compatibility optics↔NIC ↔ switch has to be verified: Cisco only accepts Cisco-coded optics on its switches (with some tolerance), Mellanox is more open.
3 · Available switch port
We check that the switch port supports the target speed. Breakout cables (1× QSFP28 100G → 4× SFP28 25G) are a useful trick for expanding 25G port capacity out of a 100G switch.
4 · Drivers and OS
Modern NIC drivers (Mellanox OFED, Intel ICE) are natively supported on recent Linux kernels, Windows Server 2019+ and ESXi 7+. For RDMA on Windows: SMB Direct is native. On Linux, RoCE / iWARP drivers have to be loaded.
5 · MTU configuration
Jumbo frames (MTU 9000) deliver better throughput on backup and replication. They must be configured consistently end-to-end: NIC, switch, remote end. A mismatch causes dropped packets and problematic path-MTU discovery.
6 · NUMA awareness
On multi-socket servers the NIC is attached to a specific CPU over PCIe. A workload running on another CPU pays for cross-NUMA traffic. Optimal NIC placement takes into account which CPU serves the primary workload.
NIC chips and models
What we fit, by speed and budget.
# 10G — enterprise entry level (default on many servers)
Intel X710-DA2 2× SFP+ · mature drivers · workhorse
Intel X722 Often onboard on recent motherboards
Broadcom 57414 2× SFP+ · common vendor-branded option
Mellanox ConnectX-4 Lx 25G capable, 10G mode
# 25G — the 2026 sweet spot
Mellanox ConnectX-5 Ex 2× SFP28 · RDMA RoCE · golden
Intel E810-XXVDA2 2× SFP28 · DPDK supported
Broadcom 57414 2× SFP28 · solid alternative
# 100G — heavy workloads / AI / NVMe-oF
Mellanox ConnectX-6 Dx 2× QSFP56 · RDMA · ML standard
Mellanox ConnectX-7 2× QSFP112 · latest gen
Intel E810-CQDA2 2× QSFP28 · enterprise generic
# OCP form factor (vendor-dedicated slot)
OCP 3.0 SFF Mellanox 25G/100G · Dell/HPE/Lenovo
OCP 3.0 SFF Intel E810 / X710 versions
# DPU / SmartNIC (for advanced cases)
NVIDIA BlueField-2/3 Network kernel offload · NVMe-oF
AMD Pensando DSC2 Programmable · SDN
How we work
Four phases.
1 · Audit of the current topology
Existing server NICs, ToR switch (model, available ports, supported speeds), current cabling, MTU configuration, any NIC teaming / LACP. We work out how the server is attached to the network before proposing an upgrade.
2 · NIC and optics selection
Target NIC based on speed, features (RDMA, DPDK) and budget. Optics / DAC consistent with the switch. We verify optics vendor compatibility with the ToR switch (Cisco/Arista/Mellanox).
3 · Installation and configuration
Physical NIC installation in the optimal slot (NUMA-aware on multi-socket), cabling, MTU configuration consistent with the switch, teaming / LACP where needed, RDMA enabled if the workload calls for it.
4 · Validation and benchmarking
iperf3 / qperf across servers to verify line rate, ping latency and that jumbo frames get through. On the real workload (backup, replication, vSAN), a synthetic test before go-live.
Anonymised real case
Hyper-V Storage Spaces Direct cluster: from 10G to 25G with RDMA.
Customer in the services sector, Brescia area: a 4-node Hyper-V cluster with hyperconverged Storage Spaces Direct (S2D). Lenovo SR650 nodes, dual-port 10G Mellanox ConnectX-4 Lx NICs. Workload: 80 production VMs (DC, file, terminal server, business applications). S2D worked over 10G, but write latency sat at 8-15 ms at idle with peaks of 30-40 ms under load — borderline for the small SQL Server database VMs.
Solution: NIC upgrade to Mellanox ConnectX-5 Ex 25G OCP 3.0 (form factor compatible with the SR650). The Mellanox SN2010 ToR switch was already 25G-capable; new SFP28 optics plus 25G inter-node DACs were required. SMB Direct (RDMA RoCE v2) enabled on the Windows Server 2022 side, jumbo frames MTU 9000 end-to-end.
Execution: four evening windows, one per node. Live migration of the VMs onto the other three nodes, server powered down, OCP NIC swap, switch port reconfiguration, boot, validation, cluster reintegration.
Outcome: S2D write latency at idle down to 1.5 ms, peaks of 4-6 ms under load. Node CPU usage noticeably lower thanks to RDMA kernel bypass. The risk profile of the SQL VMs visibly improved.
# Before · 10G ConnectX-4 Lx · no RDMA
S2D write idle 8-15 ms
S2D write peak 30-40 ms
CPU network 14-18%
Live migration ~95 sec (32GB VM)
# After · 25G ConnectX-5 Ex · RDMA RoCE
S2D write idle 1.5 ms
S2D write peak 4-6 ms
CPU network 2-3%
Live migration ~22 sec (32GB VM)
Cost drivers
Three line items on a NIC quote.
- The NIC chosen — the main variable. Certified refurbished Mellanox ConnectX-5/6 are very affordable; Intel/Broadcom are the alternatives.
- Optics and DACs — not negligible at 100G. DACs of 1-3 m are cheap; LR 10 km optics cost more.
- Switch configuration and cabling — cable replacement, MTU reconfiguration, RDMA enablement. Work coordinated with your net engineer where needed.
FAQ
The questions we get most often.
When is 1G still fine, and when does it need upgrading?
1G is enough for: small departmental servers, file servers with few users, slow traditional backup. 1G is NOT enough for: medium-to-large departmental virtualisation, vSAN/Storage Spaces (10G minimum, 25G recommended), nightly backups of terabytes within a tight window, synchronous DR replication. 10G is by now the new floor for mid-market enterprise servers.
RDMA: is it genuinely useful or just marketing?
For certain workloads, genuinely useful. RDMA (RoCE or iWARP) lets the NIC talk to the remote server's memory without involving the CPU — latency drops to 1-5 µs, throughput rises, CPU utilisation falls sharply. It is critical for: SMB Direct on Windows Server, NVMe-oF, vSAN over 25G+, GPU clustering for AI. For traditional workloads (web, mail, single-server DB) RDMA adds nothing.
Can I fit a 100G NIC in an old server?
Physically yes, if it has a free PCIe Gen3 x16 or Gen4 x8 slot. But the bottleneck simply moves to the rest of the system: the CPU cannot saturate 100G of traffic and the kernel network stack becomes the limit. Using 100G properly requires recent CPUs (Xeon Gen3+, EPYC Rome+), DPDK drivers and applications designed for offload. On older systems 25G is already a big jump, and 100G is a waste.
Do I have to replace my switches to fit 25G/100G NICs?
It depends on the switches you have. Modern ToR switches (Mellanox SN/SX, Cisco Nexus 9300, Arista 7050X) often support 25G natively or via breakout. Older 10G-only switches do not. The cost equation: 25G server NIC + new SFP28 optics + a compatible 25G switch port. The jump from 10G to 25G is often almost free because the switches are already ready.
OCP NIC vs PCIe NIC: which is the better choice?
OCP 2.0/3.0 NICs use a dedicated form factor (a proprietary internal slot) used on many recent enterprise servers for the primary NIC. Advantages: cleaner cabling, a PCIe slot freed up for other cards, integrated iLO/iDRAC management. On upgrades we prefer OCP where it is available. For additional external slots we use standard PCIe.
Which NIC brands do you recommend?
For 10G/25G enterprise: Intel X710, X722, E810 (mature drivers), Broadcom NetXtreme E-series (default on many vendors). For high-performance 25G+/100G and RDMA: Mellanox ConnectX-5/6/7 (NVIDIA, the golden standard for RDMA), Intel E810-CQDA2, Broadcom Stingray. Vendor-branded cards (Dell, HPE) are generally rebrands of Intel/Broadcom/Mellanox chips with vendor firmware.
Let's open a conversation
Tell me the make, the model and the goal. I'll come back with a plan.
Send me the brand, the model (Service Tag / Serial / motherboard part number) and the target workload. Within one working day I'll come back to you with the technical feasibility, the constraints I've spotted and an honest estimate.