After installing a new switch and moving the internet over to this , the site went offline

Quick topology refresh

Symptoms were

• FortiGate could ping anything on the internet
• Devices on the LAN could ping across the MPLS and the FortiGate without any issue.
• Devices failed to ping anything on the internet. Traceroute stopped at the FortiGate
• Debugs on the FortiGate (packet sniffer and flow debug) showed traffic successfully egressing the FortiGate.
• Disabling offloading did not fix anything.  Note hat traffic which has been offloaded to the security processor is not visible in a packet capture.

After close to 12 hours on the phone to FortiGate support with 4 L3 engineers and about to call in product engineering, we found the issue to be the FortiGate passing along some QoS information.

Quality of Services (QoS) can be applied at two different layers in the networking stack. Layer 2 QoS is known as Class of Service (CoS) and Layer 3 QoS is known as Differentiated Services Code Point (DSCP) and then there is a large amount of mapping performed in between.

CoS is marked in the 802.1q header and there are a maximum of 8 classes.

NBNCo uses CoS extensively in their network to ensure uptime and service availability (think that you want your SSH to a switch to be prioritised over someone’s youtubing). On ethernet access products (FTTP/HFC) they do allow you to use certain CoS priorities but not all and this is specific to the service configuration and if it is a TC4 or TC2 NBN service (see the difference here).

For this site the service is a FTTP (aka NFAS) and is configured using PPPoE on a VLAN tagged service (this becomes important in a moment). NBN has a very well documented network configuration for ISPs to build products out of.

https://www.nbnco.com.au/content/dam/nbnco/documents/nfas-product-technical-specification-2.0.pdf

Specifically if you look at section 6.11

If I compare a ping packet sent out the wan interface from both a the FortiGate(successful) with the Server (unsuccessful) the problem starts to become clearer.

If I look at a packet coming in from the LAN side of the FortiGate, we see the same priority being set.

And now we have a proper problem description:

FortiGate passes along QoS information from servers/workstations which is silently discarded by NBN at the NTD.

In terms of resolving the issue we have two approaches. The first is to fix the Aruba CX side, and the second is to apply this at the Fortigate side.

In FortiGate we normally use traffic shapers to mark QoS – however this is only performed at layer 3 (DSCP) and not layer 2 (QoS).

The only workaround we can apply from the FortiGate is a per policy basis on the CLI. This overrides the CoS on the frame as it flows through the Fortigate.

On the Aruba CX side, there is a decent document on QoS configuration https://www.arubanetworks.com/techdocs/AOS-CX/10.09/PDF/qos_6200-6300-6400.pdf  on Page 13.

Note this the below is the DEFAULT configuration. For some reason they’ve swapped the Best Effort and Background policy.

The fix for this should be to apply the config below. However I haven’t tested this yet and applying this change will cause ~60 seconds of disruption as QoS queues flush and rebuild.

switch(config)# qos trust cos

switch(config)# qos cos-map 1 local-priority 1

switch(config)# qos queue-profile Q1

switch(config-queue)# map queue 0 local-priority 0

switch(config-queue)# map queue 1 local-priority 1

switch(config-queue)# map queue 1 local-priority 2

switch(config-queue)# map queue 2 local-priority 3

switch(config-queue)# map queue 3 local-priority 4

switch(config-queue)# map queue 4 local-priority 5

switch(config-queue)# map queue 5 local-priority 6

switch(config-queue)# map queue 5 local-priority 7

switch(config-queue)# qos schedule-profile S1

switch(config-schedule)# dwrr queue 0 weight 5

switch(config-schedule)# dwrr queue 1 weight 10

switch(config-schedule)# dwrr queue 2 weight 15

switch(config-schedule)# dwrr queue 3 weight 20

switch(config-schedule)# dwrr queue 4 weight 25

switch(config-schedule)# dwrr queue 5 weight 50

switch(config)# apply qos queue-profile Q1 schedule-profile S1