comp.os.linux.networking

Tomek Gruca wrote:

I will try and test/post some real rules soon, but this is a bit of an
untypical setup that I haven't done before, so I'll be in
trial/error/test/repeat mode and I won't have time for a couple of days.

> I have IFB as kernel module (modprobe ifb works), bo I'm not sure how
> to test it..
> I have two classes:
> tc class add dev eth0 parent 1:0 classid 1:1 htb rate 100kbit
> tc class add dev ifb0 parent 2:0 classid 2:1 htb rate 100kbit
>
> how could I make a packet to be classified to both of them?
> I'm not sure if I understand actions correctly (probably not), can't
> find any tutorial:
> Let's say the packet is maked by iptables with mark=1001 in FORWARD
> tc filter add dev eth0 protocol ip handle 1001 fw flowid 1:1 \
> action ipt -j MARK --set-mark 2001 \
> action mirred egress redirect dev ifb0
> tc filter add dev ifb0 protocol ip handle 2001 fw flowid 2:1
> Does it make sense? The first rule classifies it to 1:1 and passes on?

I am guessing that you are doing nat on the wan interfaces, if so this
complicates things somewhat. If you are not doing nat then the easy
solution would be to direct the packets to ifb as soon as they arrive at
the wan nic(s). If you are doing nat and have the need to seperate users
then this will not work because you won't be able to see the local IP
addresses or use any iptables rules to aid your classification.

So assuming this, you are going to have to try and do everything as the
packets leave you lan facing interface - this is a bit harder as in
order to prevent loops, packets redirected to ifb when about to leave an
interface can't directly be reclassified by another qdisc on that
interface.

What should work is to classify them before the redirect and then when
they return from ifb I beleive any previous classification will be
reinstated. So if you both classify and mark it may be possible to use
the marks on the two ifbs for the link specific qdiscs and the
classification for the third qdisc on the real lan facing interface.

>
>> Depending on exactly how complicated you need your classification/qos
>> policied to be, one option I would consider is to just use policers.
>>
>
> Well... I've always used HTB for fair per user distribution of
> bandwidth..
> Don't know anything about policers but gonna read sth ;)

Policers don't queue packets they just drop or let pass depending on
whether the configured rate is exceeded (oversimplification). To make
this work they keep a virtual buffer whose length is set with the burst
parameter.

They do have a little used feature called shared meters which allows for
more complicated setups. Personally I think shared meter is a bit of a
misnomer shared policer would probably be better as they don't allow
different actions for different traffic that hits them. You may be able
to do a simple CIR with them, but on 2.6 kernels when used on ingress
they will see the traffic before it's been de-natted. On 2.4s they do
hook after de-nat. You can use them on egress aswell as ingress. So you
may be able to use them for your setup with ifb.


>
>> Shaping from the wrong end of a wan is never perfect and you need to
>> sacrifice some bandwidth whether you police or shape. The nice thing
>> about policers is that they don't add latency.

You have 15 & 6mbit that's not too bad. The less bandwidth you have the
harder things get when shaping from the wrong end of the bottleneck.

Take your 6/6 link as an example. For traffic headed from you to the wan
you have near enough total control - you put a qdisc on the interface
and set it to 6mbit and you have on your box any queue that builds up.
Your classification can do fairness and allow any interactive packets to
skip past the lower prio packets so they don't get delayed.

So what about traffic coming from the wan to your net - you can catch it
somewhere, but what rate do you set on the qdisc. it's never going to
come in faster than 6mbit so if you set 6mbit as the rate nothing will
happen, you will never build up a queue to be able to do fairness/prio
for some.

Exactly what happens will depend on what your ISP does - there may be a
big tail drop buffer that fills up so latency for all traffic suffers,
or maybe they police rate so latency is not so bad but you will get a
bit of loss and will be unable to do fairness. The way tcp works means
it will try and fill any buffer at the far end of your wan.

In practice what you do is have to compromise by setting a rate somewhat
lower than link speed and understanding that say at 80% the qdisc is
only going to fill up slowly as in the time 10 packets come in at 6meg
you will have dequeued 8. The way tcp slowstart works means that until
you get loss it will ramp up speed - so there is an incentive to use
shortish queues, but if you set them too short you hurt throughput, and
if you have lots of classes you can't be too mean to each class yet if
they are all active the length may add up to a queue that's a bit too
long. ESFQ can be handy here, but requires patching currently - sfq is
currently being enhanced aswell, but it's not mainstream yet.

Policers have another advantage for ingress - if you buffer then drop
it's a tail drop, so the tcp sender doesn't know to backoff until the
packets ahead in the queue have been dequeued - if in slowstart the
speed will still be rising and your ISP buffer still filling.

All of the above is a bit over simplistic but should give an idea of the
problems. As I said the lower the bandwidth the worse it is and you have
a reasonable amount, so all is not lost - even at lower rates it's
alot better than doing nothing, but it does mean sacrificing some
bandwidth and dropping packets that have already come in on your line.

Andy.