Configuring QoS Policer at the Provider Edge router

I'm going to configure a policer to restrict and meter the traffic coming from Customer 1 on one of the PE router. The topology I'm using is the same I've used in the MPLS L2VPN post. Except this time I configured Customer 1 in a L3VPN environment (we're running OSPF between the CE and the PE). This doesn't really affect the configuration of the Policer, the main point is that we want to limit the traffic flow rate coming from the customer (the a value usually defined in the SLA) and on the other side we usually run a shaper with the same CIR. So let's take a look how we can configure a Single-Rate Three-Color (srTCM) as well as a Two-Rate Three Color Policer (trTCM).

To configure a QoS Policer we use the MQC syntax: First we need to identify the traffic we want to police. In this example I only want to police the ICMP traffic sourced from the CE1's loopback (1.9.9.9) destined to the loopback of CE3 (1.11.11.11, the routing is already established between the two sites using OSPF). For this purpose we use the following ACL:

R7_PE1(config)#ip access-list extended PING_FROM_THE_CUSTOMER
R7_PE1(config-ext-nacl)#permit icmp host 1.9.9.9 host 1.11.11.11

We create a class-map matching the ACL we've just created:

R7_PE1(config)#class-map CLASS1
R7_PE1(config-cmap)#match access-group name PING_FROM_THE_CUSTOMER

Here we have many options to match on (CoS/DSCP for instance), and we can also use NBAR (match protocol ...) to inspect the packets up to Layer 7, but that would usually hit the CPU as well. And finally we create the policy-map:

R7_PE1(config)#policy-map POLICE_PING
R7_PE1(config-pmap)#class CLASS1

To match everything simply use the class-default class, this is the implicit "match everything" similarly to the implicit deny at the end of the ACLs. We police the incoming traffic rate using the following commands:

R7_PE1(config-pmap-c)#police 8000 bc 2000 be 3000
R7_PE1(config-pmap-c-police)#conform-action transmit 
R7_PE1(config-pmap-c-police)#exceed-action set-?
set-clp-transmit               set-cos-inner-transmit
set-cos-transmit               set-discard-class-transmit
set-dscp-transmit              set-dscp-tunnel-transmit
set-frde-transmit              set-mpls-exp-imposition-transmit
set-mpls-exp-topmost-transmit  set-prec-transmit
set-prec-tunnel-transmit       set-qos-transmit
R7_PE1(config-pmap-c-police)#exceed-action set-dscp-transmit cs1
R7_PE1(config-pmap-c-police)#violate-action drop

The incoming rate of the traffic is compared to the CIR which is 8000 bits/s in this example. The Bc (Committed Burst) defines the Tc (Committed Time Interval): Tc = (Bc [bits] / CIR [bps]) x 1000 (we cannot configure the Tc explicitly). Configuring the Bc and Be values is optional: if we miss them router assigns a value automatically to the configured CIR. Modern IOS usually tends to use Tc of 4 ms by default so it configures the Bc according to that, and the Be equals Bc by default (if Bc = Be we have a Two-Color Policer). The Bc value specifies the maximum size of the token bucket (measured in BYTES).

If the traffic is under the Bc we take the conform-action: we simply transmit the packets as they are. If the traffic is over Bc but under Be we take the exceed-action: in this example we remark the packets to CS1. If the traffic is over the Be we take the violate-action: we drop the packets.

That's everything we need for the policer, let's apply it to the customer facing interface inbound:

R7_PE1(config)#int g0/2
R7_PE1(config-if)#service-policy input POLICE_PING

Now let's verify: first I issue a ping using the physical interface:

R9_CE1#ping 1.11.11.11 repeat 100 tos 160 size 1400       
Type escape sequence to abort.
Sending 100, 1400-byte ICMP Echos to 1.11.11.11, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (100/100), round-trip min/avg/max = 3/4/6 ms

This is not affected by the policer (traffic doesn't match the ACL). None of the packets have been dropped. I also sent the pings with the CS5 DSCP marking: so I set the ToS byte to 160. How did I come up with this value? It's not necessarily straightforward but this picture helps explain:

You need to provide this value in decimal for IOS and don't forget that the ECN bits are also included (since they are part of the ToS byte), so put two trailing 0s to the end. Then convert the ToS byte to decimal (128 + 32 = 160 in this example). Now send the same packets but sourcing from the loopback:

R9_CE1#ping 1.11.11.11 repeat 100 tos 160 size 1400 source lo0
Type escape sequence to abort.
Sending 100, 1400-byte ICMP Echos to 1.11.11.11, timeout is 2 seconds:
Packet sent with a source address of 1.9.9.9 
!!!.!.!.!!.!.!!.!.!!.!.!.!!.!.!!.!.!.!!.!.!!.!.!!.!.!.!!.!.!!.!.!.!!.!
.!!.!.!!.!.!.!!.!.!!.!.!!.!.!.
Success rate is 59 percent (59/100), round-trip min/avg/max = 3/3/5 ms

Now we have a few dropped packets. Also if we take a look at the packet capture at the other side of Customer 1 (1.11.11.11) we can see that some packets arrived with the original CS5 marking (conformed packets), but many packets have been marked down to CS1. Also take a look at the sequence numbers: Customer 1 at the other side can be sure that some packets have been dropped.

Since we use MPLS in the core it's also very important to note that the Provider Edge router copies the QoS marking into the EXP bits of the MPLS header: the first 3 bits of the ToS byte (IPP) are copied to the EXP field of the MPLS header. This packet was captured between R1 and R3 (the transport label is 304 which was generated by R3) and this packet has also been remarked to CS1 at PE1 since the EXP bits are 001.

We can also explicitly mark the packets using just the MPLS EXP field. Take a look at the following example:

policy-map POLICE_PING
 class CLASS1
  police cir 8000 bc 2000 be 3000
   conform-action set-mpls-exp-imposition-transmit 2
   exceed-action set-mpls-exp-imposition-transmit 0
   violate-action drop 

This only remarks the MPLS EXP field. The DSCP field of the IP header is unchanged: Customer 1 set it to CS5 while the service provider set the MPLS EXP field to 0 in both the transport as well as in the VPN label, so this packet in the picture was exceeding.

Now let's take a look at the Two-rate Three-color Policer:

This policer uses the predefined rates (CIR and PIR), it doesn't just rely on the Bc bucket: the traffic rate for the exceed and violate actions are more predictable. How this policer works is also a bit different from the srTCM. First the incoming traffic rate is compared against the PIR (Peak Information Rate): if it's over the PIR we take the violate-action if not we compare it further to the CIR. If the traffic flow rate is above CIR but below PIR we take the exceed-action, otherwise we conform if we are below the CIR. We also have two buckets (Bc and Be), but this time the Be bucket is only filled with PIR tokens, it's not filled with tokens from the Bc bucket.

The configuration of the policy-map is very similar to the srTCM except that we have a pir value as well:

R7_PE1(config)#policy-map TWO_RATE_POLICER
R7_PE1(config-pmap)#class CLASS1
R7_PE1(config-pmap-c)#police cir 8000 bc 2000 pir 12000 be 4000
R7_PE1(config-pmap-c-police)#conform-action transmit 
R7_PE1(config-pmap-c-police)#exceed-action set-dscp-transmit cs1 
R7_PE1(config-pmap-c-police)#violate-action drop

I did the same as with the srTCM: exceed packets are marked down to CS1 and packets above the PIR (12 kbit/s) are dropped. We apply the policy-map the same way as before inbound on PE1:

R7_PE1(config)#int g0/2
R7_PE1(config-if)#service-policy input TWO_RATE_POLICER

Let's test it: first we send ICMP echoes using the physical interface as the source:

R9_CE1#ping 1.11.11.11 tos 160 size 1400 repeat 100
Type escape sequence to abort.
Sending 100, 1400-byte ICMP Echos to 1.11.11.11, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (100/100), round-trip min/avg/max = 3/3/6 ms

As expected the policer didn't take effect: all packets are transmitted successfully. Now we match CLASS1 using the loopback address as the source:

R9_CE1#ping 1.11.11.11 tos 160 size 1400 repeat 100 source lo0
Type escape sequence to abort.
Sending 100, 1400-byte ICMP Echos to 1.11.11.11, timeout is 2 seconds:
Packet sent with a source address of 1.9.9.9 
!!.!!.!!.!!.!!.!!.!!.!!.!!.!!.!!.!!.!!.!!.!!.!!.!!.!!.!!.!!.!!.!!.!!.!
!.!!.!!.!!.!!.!!.!!.!!.!!.!!.!
Success rate is 67 percent (67/100), round-trip min/avg/max = 2/3/6 ms

The traffic is policed to 12 kbit/s and we have many dropped ICMP echoes. I'd say the traffic pattern is much more predictable: the violate-action relies on the PIR value and not on the tokens filled from the Bc to the Be bucket.

Now what can we do on the Customer side to prevent the drops? We can shape the traffic rate to the same value where the Service Provider polices the traffic: in this example let's shape the traffic rate to 8 kbit/s on the Customer side:

R9_CE1(config)#policy-map SHAPE
R9_CE1(config-pmap)#class class-default
R9_CE1(config-pmap-c)#shape average 8000

I use the class-default class here to match everything, and we apply the shaper outbound on the service provider facing interface:

R9_CE1(config)#int g0/0
R9_CE1(config-if)#service-policy output SHAPE

And let's run the same ping again:

R9_CE1#ping 1.11.11.11 source lo0 repeat 100 size 1400
Type escape sequence to abort.
Sending 100, 1400-byte ICMP Echos to 1.11.11.11, timeout is 2 seconds:
Packet sent with a source address of 1.9.9.9 
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (100/100), round-trip min/avg/max = 4/1429/1908 ms

Now we have 0 dropped packets and 100% success rate, but look at the average RTT: it's more than 1400 ms (!) which is unacceptable in a real-world environment. So the shaper can introduce a lot of delay since it needs to buffer the packets in memory before putting the packets onto the transmit ring.