Tomasz Wendlandt

Avatar

Tech blog

Transparent spreading of incoming network traffic load across CPUs

TuxCiekawy ficzer w kernelu 2.6.35.

Network cards have improved the bandwidth to the point where it’s hard for a single modern CPU to keep up. Two new features contributed by Google aim to spread the load of network handling across the CPUs available in the system: Receive Packet Steering (RPS) and Receive Flow Steering (RFS).

RPS distributes the load of received packet processing across multiple CPUs. This solution allows protocol processing (e.g. IP and TCP) to be performed on packets in parallel (contrary to the previous code). For each device (or each receive queue in a multi-queue device) a hashing of the packet header is used to index into a mask of CPUs (which can be configured manually in /sys/class/net//queues/rx-/rps_cpus) and decide which CPU will be used to process a packet. But there’re also some heuristics provided by the RFS side of this feature. Instead of randomly choosing the CPU from a hash, RFS tries to use the CPU where the application running the recvmsg() syscall is running or has run in the past, to improve cache utilization. Hardware hashing is used if available. This feature effectively emulates what a multi-queue NIC can provide, but instead it is implement in software and for all kind of network hardware, including single queue cards and not excluding multiqueue cards.

Benchmarks of 500 instances of netperf TCP_RR test with 1 byte request and response show the potential benefit of this feature, a e1000e network card on 8 core Intel server goes from 104K tps at 30% CPU usage, to 303K tps at 61% CPU usage when using RPS+RFS. A RPC test which is similar in structure to the netperf RR test with 100 threads on each host, but doing more work in userspace that netperf, goes from 103K tps at 48% of CPU utilization to 223K at 73% CPU utilization and much lower latency.

Więcej o RPS i RFS:
http://lwn.net/Articles/362339/
http://lwn.net/Articles/362339/
http://www.kernel.org/pub/linux/kernel/v2.6/ChangeLog-2.6.35

Źródło

No Comments, Comment or Ping

Reply to “Transparent spreading of incoming network traffic load across CPUs”

Poznań

  • Light Rain
  • Temperature: 5°C
  • Visibility: 10km
  • Conditions: Light Rain
  • Clouds: Broken Clouds at 914m
  • Wind: SW at 20 km/h
  • Barometer: 1011 hPa
  • Humidity: 80.9%
  • Sunrise: 8:10 GMT+2
  • Sunset: 20:17 GMT+2