Solaris, ipfilter accounting and tagged interfaces

This is a copy of a posting I made to the LBW-mailinglist. Maybe you got some ideas to share. Thanks!

Hi guys,
I got a problem which is really driving me nuts. I got this huge farm
of proxies running Solaris 10, shoving around a dozen gigabits/s at my
customer.
Ingress and Egress traffic run over two different VLANs, so I
configured two tagged interfaces on one physical. Also, there is a
redundant interface with the same config with IPMP over it. It looks
like...

                   +--- egress IPMP group ----+
                   |                          |
                   (                          |
      +--- ingress IPMP group ---+            |
      |            (             |            |
      |            |             |            |
+-----+-----+------+-----+ +-----+-----+------+-----+
| bge111000 | bge1112000 | | bnx111000 | bnx1112000 |
|  ingress  |   egress   | |  ingress  |   egress   |
+------------------------+ +------------------------+
|          bge0          | |          bnx0          |
+------------------------+ +------------------------+

To observe traffic on all those proxy-nodes I let Cacti (fancy
rrdtool graphics) poll the nodes' SNMP-agent and leech off the traffic
statistics from the tagged interfaces (bge111000 and so on).

This worked fine until Solaris 10 05/07, when they introduced a bug in
the bge-NIC-drivers. Polling bge111000 and bge111100 didn't result in
individual statistiscs, but both interfaces show the accumulated
traffic from the underlying physical bge0. Netstat -i shows the same,
so it must be the bge-Driver. Well, we complained at Sun and got the
answer that the next release will fix the problem. Since we used IPMP
anyway, we were able to set the active interfaces to the bnx-NICs,
whose drivers were not affected by the problem

Now we upgraded to Sol10 5/09 and guess what: The bge-drivers where
NOT fixed, but now the bnx-Drivers are screwed up too!

I thought about alternatives:
* Cisco can't differentiate on Trunks too.
* Netflow was no option, because our access-switches don't support it.
* Let's use ipfilter and ip-accounting!

So I set up a lot of accounting-rules (VLAN-ids not the same here, but
you get the idea:

# Ingress
count in on bge923000 from any to any
count in on bnx923000 from any to any
count in on bge924000 from any to any
count in on bnx924000 from any to any
# Egress
count out on bge923000 from any to any
count out on bnx923000 from any to any
count out on bge924000 from any to any
count out on bnx924000 from any to any

So guess what happens... NUTHIN:
# ipfstat -ai
0 count in on bge923000 from any to any
0 count in on bnx923000 from any to any
0 count in on bge924000 from any to any
0 count in on bnx924000 from any to any
# ipfstat -ao
0 count out on bge923000 from any to any
0 count out on bnx923000 from any to any
0 count out on bge924000 from any to any
0 count out on bnx924000 from any to any

Compared to a real untagged interface:
# ipfstat -ai
1724 count in on bge1 from any to any
# ipfstat -ao
1864 count out on bge1 from any to any

So. Sorry for the long posting.
1) Am I stupid or just cursed?
2) How can I get those dang per-VLAN-statistics on my nodes?
3) Does anyone know if ipfilter on Solaris supports tagged interfaces at all?
4) How is this going to end?

Oh Eris, I really need a stiff drink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: