{"id":6110,"date":"2022-05-04T07:13:56","date_gmt":"2022-05-04T07:13:56","guid":{"rendered":"https:\/\/pariswells.com\/blog\/?p=6110"},"modified":"2022-09-04T07:16:03","modified_gmt":"2022-09-04T07:16:03","slug":"qos-nbn-fttp-fortigate-and-aruba-cx-switches","status":"publish","type":"post","link":"https:\/\/pariswells.com\/blog\/research\/qos-nbn-fttp-fortigate-and-aruba-cx-switches","title":{"rendered":"QoS, NBN FTTP, FortiGate and Aruba CX Switches"},"content":{"rendered":"<p class=\"x_MsoNormal\">After installing a new switch and moving the internet over to this , the site went offline<\/p><p class=\"x_MsoNormal\" aria-hidden=\"true\">\u00a0<\/p><p class=\"x_MsoNormal\">Quick topology refresh<\/p><p><img loading=\"lazy\" decoding=\"async\" width=\"246\" height=\"136\" class=\"alignnone size-full wp-image-6111  img-responsive\" src=\"http:\/\/pariswells.com\/blog\/wp-content\/uploads\/2022\/09\/img_6314503a44bb3.png\" alt=\"\" \/><\/p><p class=\"x_MsoNormal\" aria-hidden=\"true\">\u00a0<\/p><p class=\"x_MsoNormal\">Symptoms were<\/p><ul type=\"disc\"><li class=\"x_MsoListParagraph\">FortiGate could ping anything on the internet<\/li><li class=\"x_MsoListParagraph\">Devices on the LAN could ping across the MPLS and the FortiGate without any issue.<\/li><li class=\"x_MsoListParagraph\">Devices failed to ping anything on the internet. Traceroute stopped at the FortiGate<\/li><li class=\"x_MsoListParagraph\">Debugs on the FortiGate (packet sniffer and flow debug) showed traffic successfully egressing the FortiGate.<\/li><li class=\"x_MsoListParagraph\">Disabling offloading did not fix anything.\u00a0 Note hat traffic which has been offloaded to the security processor is not visible in a packet capture.<\/li><\/ul><p class=\"x_MsoNormal\" aria-hidden=\"true\">\u00a0<\/p><p class=\"x_MsoNormal\">After close to 12 hours on the phone to FortiGate support with 4 L3 engineers and about to call in product engineering, we found the issue to be the FortiGate passing along some QoS information.<\/p><p class=\"x_MsoNormal\" aria-hidden=\"true\">\u00a0<\/p><p class=\"x_MsoNormal\">Quality of Services (QoS) can be applied at two different layers in the networking stack. Layer 2 QoS is known as Class of Service (CoS) and Layer 3 QoS is known as Differentiated Services Code Point (DSCP) and then there is a large amount of mapping performed in between.<\/p><p class=\"x_MsoNormal\" aria-hidden=\"true\">\u00a0<\/p><p class=\"x_MsoNormal\">CoS is marked in the 802.1q header and there are a maximum of 8 classes.<\/p><p><img loading=\"lazy\" decoding=\"async\" width=\"629\" height=\"207\" class=\"alignnone size-full wp-image-6113  img-responsive\" src=\"http:\/\/pariswells.com\/blog\/wp-content\/uploads\/2022\/09\/img_6314503b154d2.png\" alt=\"\" srcset=\"https:\/\/pariswells.com\/blog\/wp-content\/uploads\/2022\/09\/img_6314503b154d2.png 629w, https:\/\/pariswells.com\/blog\/wp-content\/uploads\/2022\/09\/img_6314503b154d2-300x99.png 300w\" sizes=\"auto, (max-width: 629px) 100vw, 629px\" \/><\/p><p class=\"x_MsoNormal\" aria-hidden=\"true\">\u00a0<\/p><p class=\"x_MsoNormal\">NBNCo uses CoS extensively in their network to ensure uptime and service availability (think that you want your SSH to a switch to be prioritised over someone\u2019s youtubing). On ethernet access products (FTTP\/HFC) they do allow you to use certain CoS priorities but not all and this is specific to the service configuration and if it is a TC4 or TC2 NBN service (see the difference\u00a0<a href=\"https:\/\/www.aussiebroadband.com.au\/help-centre\/business\/enterprise\/what-is-nbn-tc-4-and-tc-2\/\" target=\"_blank\" rel=\"noopener noreferrer\" data-auth=\"NotApplicable\" data-linkindex=\"0\">here<\/a>).<\/p><p class=\"x_MsoNormal\" aria-hidden=\"true\">\u00a0<\/p><p class=\"x_MsoNormal\">For this site the service is a FTTP (aka NFAS) and is configured using PPPoE on a VLAN\u00a0<b><u>tagged<\/u><\/b>\u00a0service (this becomes important in a moment). NBN has a very well documented network configuration for ISPs to build products out of.<\/p><p class=\"x_MsoNormal\"><a href=\"https:\/\/www.nbnco.com.au\/content\/dam\/nbnco\/documents\/nfas-product-technical-specification-2.0.pdf\" target=\"_blank\" rel=\"noopener noreferrer\" data-auth=\"NotApplicable\" data-linkindex=\"1\">https:\/\/www.nbnco.com.au\/content\/dam\/nbnco\/documents\/nfas-product-technical-specification-2.0.pdf<\/a><\/p><p class=\"x_MsoNormal\">Specifically if you look at section 6.11<\/p><p><img loading=\"lazy\" decoding=\"async\" width=\"656\" height=\"242\" class=\"alignnone size-full wp-image-6114  img-responsive\" src=\"http:\/\/pariswells.com\/blog\/wp-content\/uploads\/2022\/09\/img_6314503b43797.png\" alt=\"\" srcset=\"https:\/\/pariswells.com\/blog\/wp-content\/uploads\/2022\/09\/img_6314503b43797.png 656w, https:\/\/pariswells.com\/blog\/wp-content\/uploads\/2022\/09\/img_6314503b43797-300x111.png 300w\" sizes=\"auto, (max-width: 656px) 100vw, 656px\" \/><\/p><p class=\"x_MsoNormal\" aria-hidden=\"true\">\u00a0<\/p><p class=\"x_MsoNormal\">If I compare a ping packet sent out the wan interface from both a the FortiGate(successful) with the Server (unsuccessful) the problem starts to become clearer.<\/p><p><img loading=\"lazy\" decoding=\"async\" width=\"1101\" height=\"348\" class=\"alignnone size-full wp-image-6116  img-responsive\" src=\"http:\/\/pariswells.com\/blog\/wp-content\/uploads\/2022\/09\/img_6314503b93391.png\" alt=\"\" srcset=\"https:\/\/pariswells.com\/blog\/wp-content\/uploads\/2022\/09\/img_6314503b93391.png 1101w, https:\/\/pariswells.com\/blog\/wp-content\/uploads\/2022\/09\/img_6314503b93391-300x95.png 300w, https:\/\/pariswells.com\/blog\/wp-content\/uploads\/2022\/09\/img_6314503b93391-1024x324.png 1024w, https:\/\/pariswells.com\/blog\/wp-content\/uploads\/2022\/09\/img_6314503b93391-768x243.png 768w\" sizes=\"auto, (max-width: 1101px) 100vw, 1101px\" \/><\/p><p class=\"x_MsoNormal\" aria-hidden=\"true\">\u00a0<\/p><p class=\"x_MsoNormal\">If I look at a packet coming in from the LAN side of the FortiGate, we see the same priority being set.<\/p><p><img loading=\"lazy\" decoding=\"async\" width=\"655\" height=\"179\" class=\"alignnone size-full wp-image-6112  img-responsive\" src=\"http:\/\/pariswells.com\/blog\/wp-content\/uploads\/2022\/09\/img_6314503a9ac12.png\" alt=\"\" srcset=\"https:\/\/pariswells.com\/blog\/wp-content\/uploads\/2022\/09\/img_6314503a9ac12.png 655w, https:\/\/pariswells.com\/blog\/wp-content\/uploads\/2022\/09\/img_6314503a9ac12-300x82.png 300w\" sizes=\"auto, (max-width: 655px) 100vw, 655px\" \/><\/p><p class=\"x_MsoNormal\" aria-hidden=\"true\">\u00a0<\/p><p class=\"x_MsoNormal\">And now we have a proper problem description:<\/p><p class=\"x_MsoNormal\">FortiGate passes along QoS information from servers\/workstations which is silently discarded by NBN at the NTD.<\/p><p class=\"x_MsoNormal\">In terms of resolving the issue we have two approaches. The first is to fix the Aruba CX side, and the second is to apply this at the Fortigate side.<\/p><p class=\"x_MsoNormal\">In FortiGate we normally use traffic shapers to mark QoS \u2013 however this is only performed at layer 3 (DSCP) and not layer 2 (QoS).<\/p><p class=\"x_MsoNormal\">The only workaround we can apply from the FortiGate is a\u00a0<b>per policy<\/b>\u00a0basis on the CLI. This overrides the CoS on the frame as it flows through the Fortigate.<\/p><div><p class=\"x_MsoNormal\">config\u00a0firewall\u00a0policy<br aria-hidden=\"true\" \/>\u00a0\u00a0\u00a0\u00a0edit\u00a034<br aria-hidden=\"true\" \/>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0set\u00a0vlan-cos-fwd\u00a00<br aria-hidden=\"true\" \/>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0set\u00a0vlan-cos-rev\u00a00<br aria-hidden=\"true\" \/>\u00a0\u00a0\u00a0\u00a0next<br aria-hidden=\"true\" \/>end<\/p><\/div><p class=\"x_MsoNormal\" aria-hidden=\"true\">\u00a0<\/p><p class=\"x_MsoNormal\">On the Aruba CX side, there is a decent document on QoS configuration\u00a0<a href=\"https:\/\/www.arubanetworks.com\/techdocs\/AOS-CX\/10.09\/PDF\/qos_6200-6300-6400.pdf\" target=\"_blank\" rel=\"noopener noreferrer\" data-auth=\"NotApplicable\" data-linkindex=\"2\">https:\/\/www.arubanetworks.com\/techdocs\/AOS-CX\/10.09\/PDF\/qos_6200-6300-6400.pdf<\/a>\u00a0\u00a0on Page 13.<\/p><p class=\"x_MsoNormal\">Note this the below is the DEFAULT configuration. For some reason they\u2019ve swapped the Best Effort and Background policy.<\/p><p class=\"x_MsoNormal\" aria-hidden=\"true\">\u00a0<\/p><p><img loading=\"lazy\" decoding=\"async\" width=\"892\" height=\"265\" class=\"alignnone size-full wp-image-6115  img-responsive\" src=\"http:\/\/pariswells.com\/blog\/wp-content\/uploads\/2022\/09\/img_6314503b64a74.png\" alt=\"\" srcset=\"https:\/\/pariswells.com\/blog\/wp-content\/uploads\/2022\/09\/img_6314503b64a74.png 892w, https:\/\/pariswells.com\/blog\/wp-content\/uploads\/2022\/09\/img_6314503b64a74-300x89.png 300w, https:\/\/pariswells.com\/blog\/wp-content\/uploads\/2022\/09\/img_6314503b64a74-768x228.png 768w\" sizes=\"auto, (max-width: 892px) 100vw, 892px\" \/><\/p><p class=\"x_MsoNormal\" aria-hidden=\"true\">\u00a0<\/p><p class=\"x_MsoNormal\">The fix for this should be to apply the config below. However I haven\u2019t tested this yet and applying this change will cause ~60 seconds of disruption as QoS queues flush and rebuild.<\/p><p class=\"x_MsoNormal\" aria-hidden=\"true\">\u00a0<\/p><p class=\"x_MsoNormal\">switch(config)# qos trust cos<\/p><p class=\"x_MsoNormal\">switch(config)# qos cos-map 1 local-priority 1<\/p><p class=\"x_MsoNormal\">switch(config)# qos queue-profile Q1<\/p><p class=\"x_MsoNormal\">switch(config-queue)# map queue 0 local-priority 0<\/p><p class=\"x_MsoNormal\">switch(config-queue)# map queue 1 local-priority 1<\/p><p class=\"x_MsoNormal\">switch(config-queue)# map queue 1 local-priority 2<\/p><p class=\"x_MsoNormal\">switch(config-queue)# map queue 2 local-priority 3<\/p><p class=\"x_MsoNormal\">switch(config-queue)# map queue 3 local-priority 4<\/p><p class=\"x_MsoNormal\">switch(config-queue)# map queue 4 local-priority 5<\/p><p class=\"x_MsoNormal\">switch(config-queue)# map queue 5 local-priority 6<\/p><p class=\"x_MsoNormal\">switch(config-queue)# map queue 5 local-priority 7<\/p><p class=\"x_MsoNormal\">switch(config-queue)# qos schedule-profile S1<\/p><p class=\"x_MsoNormal\">switch(config-schedule)# dwrr queue 0 weight 5<\/p><p class=\"x_MsoNormal\">switch(config-schedule)# dwrr queue 1 weight 10<\/p><p class=\"x_MsoNormal\">switch(config-schedule)# dwrr queue 2 weight 15<\/p><p class=\"x_MsoNormal\">switch(config-schedule)# dwrr queue 3 weight 20<\/p><p class=\"x_MsoNormal\">switch(config-schedule)# dwrr queue 4 weight 25<\/p><p class=\"x_MsoNormal\">switch(config-schedule)# dwrr queue 5 weight 50<\/p><p class=\"x_MsoNormal\">switch(config)# apply qos queue-profile Q1 schedule-profile S1<\/p>","protected":false},"excerpt":{"rendered":"<p>After installing a new switch and moving the internet over to this , the site went offline\u00a0Quick topology refresh\u00a0Symptoms wereFortiGate could ping anything on the internetDevices on [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[2676,1251,3944,3945,3943,1909,3946,3947],"class_list":["post-6110","post","type-post","status-publish","format-standard","hentry","category-research","tag-aruba","tag-fortigate","tag-nbn","tag-nbnco","tag-qos","tag-vlan","tag-vlan-cos-fwd","tag-vlan-cos-rev"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/pariswells.com\/blog\/wp-json\/wp\/v2\/posts\/6110","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/pariswells.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/pariswells.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/pariswells.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/pariswells.com\/blog\/wp-json\/wp\/v2\/comments?post=6110"}],"version-history":[{"count":1,"href":"https:\/\/pariswells.com\/blog\/wp-json\/wp\/v2\/posts\/6110\/revisions"}],"predecessor-version":[{"id":6117,"href":"https:\/\/pariswells.com\/blog\/wp-json\/wp\/v2\/posts\/6110\/revisions\/6117"}],"wp:attachment":[{"href":"https:\/\/pariswells.com\/blog\/wp-json\/wp\/v2\/media?parent=6110"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/pariswells.com\/blog\/wp-json\/wp\/v2\/categories?post=6110"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/pariswells.com\/blog\/wp-json\/wp\/v2\/tags?post=6110"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}