> ISP throttles upstream DSCP CS1 traffic

Live speed test

Summary

The ISP slows all upstream packets that have the DSCP mark CS1. The limit is approximately 1–2 Mbps. The ISP also slows the mark AF12 to approximately 25 Mbps. Other marks are not affected. Downloads are not affected.

OpenSSH on Linux uses the mark CS1 for bulk transfers. Thus, scp, rsync and other SSH uploads from the LAN were almost stopped.

The gateway now clears all DSCP marks on the WAN port. A timer on the gateway applies this setting again after each reboot. The Linux hosts also do not set DSCP marks on SSH traffic now. CS1 uploads operate at the full line rate.

Symptom

We sent the same upload with different DSCP marks. We sent each test in the same minute. Only the mark changed.

The limit applies to exact DSCP values. DSCP 9 is adjacent to CS1, but it operates at full speed. Thus, the cause is a classifier that looks for specific values.

It is a shaper, not a drop

CS1 pings to 1.1.1.1 had 0% loss. But large CS1 pings had more delay:

The added delay is approximately 10 ms for one 1428-byte packet. This agrees with a queue that drains at approximately 1 Mbps.

Fault location

We captured packets on the gateway with tcpdump. We recorded the time of each CS1 packet at each interface.

• The gateway forwards each CS1 packet in approximately 50 µs.
• The gateway does not change the mark.
• All the added delay occurs after the WAN port. Thus, the fault is in the ISP equipment or network.

Probable cause

We cannot see the ISP configuration. The data agrees with this explanation:

• CS1 is the "scavenger" mark. Applications use it to request low priority. Examples: SSH bulk transfers, some backup and sync clients, some update downloads.
• AF12 is a "high-throughput data" mark. Corporate networks and some VPN clients use it.
• Usually, an ISP ignores or clears customer DSCP marks at the network edge.
• This ISP probably uses a DSCP-to-queue table from a business or voice service. The table sends CS1 and AF12 to low-priority queues.
• A scavenger queue must yield only during congestion. This queue has a fixed low rate, also on an idle line. This is probably a configuration error.

Items that we eliminated

Workaround 1: gateway

The gateway kernel does not have a DSCP rewrite module. The iptables DSCP target, act_pedit and dsmark are all not available. The cake queue discipline is available. Its wash option clears the DSCP field on egress.

One cake instance on the WAN root limited the line to approximately 1.25 Gbps. One CPU core did all the work. Thus, we put one cake instance on each of the four transmit queues:

# one cake instance per TX queue, DSCP wash, no rate limit
# modprobe sch_cake
# tc qdisc replace dev <wan> root handle 1: mq
# for q in 1 2 3 4; do
    tc qdisc replace dev <wan> parent 1:$q cake unlimited besteffort wash
  done

# revert to the default
# tc qdisc del dev <wan> root

A packet capture shows the result. A CS1 packet enters the gateway with tos 0x20. It leaves the WAN port with tos 0x00.

Persistence

A reboot, a firmware update or a change to the WAN settings removes the queue setting. Thus, we installed a systemd timer on the gateway:

• The script is in /data, which keeps its contents after a reboot.
• The timer starts 30 s after boot. Then it starts each 60 s.
• The script examines each WAN interface. If the setting is not there, the script applies it again. It also writes a log entry.
• The script does not change a correct setting.

Workaround 2: Linux hosts

Each Linux host on the LAN now has this OpenSSH client setting:

IPQoS none

SSH then sends unmarked packets. This setting gives protection if the gateway setting is not there. The base build script for new hosts also includes it.

Verification: on/off test

We removed the workaround and then installed it again, in the same minute.

Verification: sustained load

Verification: ceiling

We used 18 parallel uploads (6 processes × 3 streams) for 45–60 s per test. The LAN host has a 2.5 GbE link.

• Default queues: 2,053–2,059 Mbps. But CS1 traffic is throttled.
• One cake instance: 1,241–1,280 Mbps. This is a CPU limit.
• Four cake instances: CS1 and unmarked traffic both reach the line rate.
• Cake dropped 7 of 43.5 million packets.
• Gateway CPU at ceiling: 81–88% busy (4 cores).
• Ping RTT at ceiling: 16–18 ms avg. With the line not full: approximately 2–5 ms.

Open items

1. Make sure that the timer is there after the next gateway firmware update. If it is not there, install it again.
2. Do a real SSH upload test from a second LAN host with -o IPQoS=cs1. Status: pending.

Decision: The owner will not send a fault report to the ISP. The workaround stays in place.

Method

• Test tool: Python sockets with IP_TOS set per connection. HTTPS POST to speed.cloudflare.com/__up.
• SSH test: ssh -o IPQoS=cs1 against ssh -o IPQoS=none, 200 MB upload.
• Latency: ICMP to 1.1.1.1 with ping -Q.
• Location: tcpdump -i any on the gateway. Per-interface timestamps.
• Gateway state: tc -s qdisc, iptables-save, /proc/stat, kernel config.