ExtremeCloud IQ- Site Engine & Extreme Management Center

Expand all | Collapse all

purview receives netflow and mirrorN, but apps are not identified

Mike Thomas

Mike Thomas11-08-2016 17:44

  • 1.  purview receives netflow and mirrorN, but apps are not identified

    Posted 11-08-2016 09:42
    Hi,
    I installed a virtual vmware purview/analytics AAE in deployment mode 2 (Interface mirrored), with eth0 for netflow and management and eth1 for mirror packet reception:
    0. Accept settings and continue
    1. Hostname: interpur
    2. Deployment Mode: Dual Interface Mirrored
    3. Management Interface Configuration (eth0):
    Address: 192.168.64.220
    Netmask: 255.255.254.0
    Gateway: 192.168.64.1
    Nameserver: [our dns]
    Domain name: [our domain]
    4. NIS Server/Domain: Not Configured
    5. Monitor Interface Configuration :
    Tap Mode Interfaces eth1
    [/code]An SSA switch sends its netflow packets towards 192.168.64.220 and has a mirror-n towards one of its switch ports which is directly connected to the AAE's host. (AAE's eth1 is mapped to a dedicated vswitch with promiscuous mode and All VLAN (4095) enabled. This vswitch uses the esxi-host's physical adpater which is directly connected to the SSA mirror port ge.1.17.)

    SSA mirror and netflowconfig:
    # mirror
    set mirror create 2
    set mirror 2 mirrorN 15
    set mirror ports ge.1.17 2
    # policy
    set policy profile 1 name Purviewmirror pvid-status enable pvid 4095 mirror-destination 2
    set policy rule admin-profile port lag.0.4 mask 16 port-string lag.0.4 admin-pid 1
    set policy rule admin-profile port lag.0.5 mask 16 port-string lag.0.5 admin-pid 1
    set policy rule admin-profile port ge.1.45 mask 16 port-string ge.1.45 admin-pid 1
    set policy rule admin-profile port tg.1.4 mask 16 port-string tg.1.4 admin-pid 1
    # netflow
    set netflow export-interval 1
    set netflow export-destination 192.168.64.220 2055
    set netflow export-version 9
    set netflow port lag.0.4-5 enable rx
    set netflow port ge.1.45 enable rx
    set netflow port tg.1.4 enable rx
    set netflow template refresh-rate 30 timeout 1
    set netflow cache enable[/code]AAE receives netflow packets on eth0:
    root@interpur:~$ tcpdump -i eth0 udp port 2055
    tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
    listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
    11:28:35.047775 IP 192.168.64.9.2055 > interpur.man.htw-berlin.de.2055: UDP, length 1420
    11:28:35.052489 IP 192.168.64.9.2055 > interpur.man.htw-berlin.de.2055: UDP, length 1444
    11:28:35.058061 IP 192.168.64.9.2055 > interpur.man.htw-berlin.de.2055: UDP, length 1464
    ...[/code]AAE receives mirror packets on eth1:
    [/code]root@interpur:~$ tcpdump -i eth1 -c 2
    tcpdump: WARNING: eth1: no IPv4 address assigned
    tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
    listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
    11:32:07.549780 20:b3:99??72:3b (oui Unknown) > 20:b3:99:55:a0:1f (oui Unknown), ethertype Unknown (0x7034), length 70:
    0x0000: a001 0800 4500 0034 01b2 4000 3406 395e ....E..4..@.4.9^
    0x0010: 40ca 7004 8d2d cdb8 0050 c52b 245a cadb @.p..-...P.+$Z..
    0x0020: 12e4 469b 8010 003e e21e 0000 0101 080a ..F....>........
    0x0030: 6b12 fa03 36f0 de74 k...6..t
    11:32:07.549789 20:b3:99??72:3b (oui Unknown) > 20:b3:99:55:a0:1f (oui Unknown), ethertype Unknown (0x7034), length 64:
    0x0000: a001 0800 4500 002c 104b 0000 3006 d058 ....E..,.K..0..X
    0x0010: bad4 81d1 8d2d e055 0aac 0913 8d2d e055 .....-.U.....-.U
    0x0020: 0000 0000 6002 11b4 5b0f 0000 0204 05ac ....'...[.......
    0x0030: 0000 ..[/code]
    Management Center also shows netflow and mirror packets:



    But the Identification Rate stays at 0% and Application Infos are not populated. What could be the reason? AAE was enforced and rebooted.



  • 2.  RE: purview receives netflow and mirrorN, but apps are not identified

    Posted 11-08-2016 10:01
    Hi, do you have enabled jumbo frames on all the path between the sensor and the engine?


  • 3.  RE: purview receives netflow and mirrorN, but apps are not identified

    Posted 11-08-2016 10:30
    Hi,
    on SSA-Switch "show port jumbo ge.1.17" shows jumbo is adminstratively enabled and operational with MTU 10239.


  • 4.  RE: purview receives netflow and mirrorN, but apps are not identified

    Posted 11-08-2016 10:33
    Ok, ensure that jumbo is enabled along all the path...then you need to wait for a while before the engine collects the flows and display it on the ExtremeControl...


  • 5.  RE: purview receives netflow and mirrorN, but apps are not identified

    Posted 11-08-2016 10:40
    Jumbo was enabled from the beginning on this switch port, so there is nothing to wait for. The path is quite short becauase the VMs tap-port vswitch is directly connected to the switch port.

    edit:
    Netsight Console Alarm says: "No bidirectional traffic seen on interface eth1". It seems as if the appliance gets mirror packets, but does not recognize them.


  • 6.  RE: purview receives netflow and mirrorN, but apps are not identified

    Posted 11-08-2016 11:10
    Hi.

    configure ge.1.17 for 1000 Full Duplex.

    set port duplex ge.1.17 full

    set port speed ge.1.17 1000

    regards

    Alex



  • 7.  RE: purview receives netflow and mirrorN, but apps are not identified

    Posted 11-08-2016 11:18
    Under Analytics, go to configuration, then to location.Make sure the "location" is set with the IP range that you are monitoring.



  • 8.  RE: purview receives netflow and mirrorN, but apps are not identified

    Posted 11-08-2016 11:29
    Hi,
    1000 full duplex was already configured.

    edit: Locations exist. (I already used a GRE-tunnel purview successfully with identified apps.)


  • 9.  RE: purview receives netflow and mirrorN, but apps are not identified

    Posted 11-08-2016 12:24
    1. In a 'show port status ge.1.17' the mirrorN port must be operationally and administratively up.
    2. The packet capture should be run longer, and make sure that the traffic seen in both directions.

    The policy and netflow portions look correct. But lets make sure that that eth1 is receiving bilateral communications from those ports.


  • 10.  RE: purview receives netflow and mirrorN, but apps are not identified

    Posted 11-08-2016 12:51
    Hi,
    >show port status ge.1.17
    Port Alias Oper Admin Speed Duplex Type
    (truncated) Status Status (bps)
    ------------ ---------------- -------- ------- ------ ------- ------------------
    ge.1.17 up up 1.0G full 1000-t rj45

    [/code]The mirror setting definitively mirrors bilateral communications. tg.1.4 and ge.1.45 are ports towards external network and lag.0.4-5 are ports to internal networks. These ports do netflow and mirror ingress packets. A GRE-tunnel purview was able to decode bilateral traffic from that n-mirror. I cant paste much longer captures here as there are about 10,000 packets per second.
    tcpdump -i eth1 -q
    14:45:03.936153 20:b3:99??72:3b (oui Unknown) > 00:16:9c:6e:2f:80 (oui Unknown), Unknown Ethertype (0x7034), length 74:
    14:45:03.936299 20:b3:99??72:3b (oui Unknown) > 20:b3:99:55:a0:1f (oui Unknown), Unknown Ethertype (0x7034), length 70:
    14:45:03.936420 20:b3:99??72:3b (oui Unknown) > 20:b3:99:55:a0:1f (oui Unknown), Unknown Ethertype (0x7034), length 64:
    14:45:03.936425 20:b3:99??72:3b (oui Unknown) > 00:16:9c:6e:2f:80 (oui Unknown), Unknown Ethertype (0x7034), length 77:
    14:45:03.936552 20:b3:99??72:3b (oui Unknown) > 20:b3:99:55:a0:1f (oui Unknown), Unknown Ethertype (0x7034), length 99:
    14:45:03.936602 20:b3:99??72:3b (oui Unknown) > 00:16:9c:6e:2f:80 (oui Unknown), Unknown Ethertype (0x7034), length 307:
    14:45:03.936728 20:b3:99??72:3b (oui Unknown) > 00:16:9c:6e:2f:80 (oui Unknown), Unknown Ethertype (0x7034), length 965:
    14:45:03.936850 20:b3:99??72:3b (oui Unknown) > 00:16:9c:6e:2f:80 (oui Unknown), Unknown Ethertype (0x7034), length 171:

    [/code]I am wondering why the dump from eth1 says "oui unknown" and "ethertype 0x7034" in EVERY packet. This doesnt look normal. Maybe something offsets the packets so AEE cant decode them?



  • 11.  RE: purview receives netflow and mirrorN, but apps are not identified

    Posted 11-08-2016 13:23
    This appears to be one way traffic - 0x7034 appears to be CDP/EDP Discovery protocol packets. So I think your just seeing passive data from attached S-Series port. I dont think data is being mirrored properly from the S-Series.
    I would stick a network analyzer on the mirror port directly (ge.1.17) to see if it is forwarding two way traffic, or if the problem is localized possibly to V-Switch.


  • 12.  RE: purview receives netflow and mirrorN, but apps are not identified

    Posted 11-08-2016 15:20
    Hi,
    This is a wireshark trace directly from the n-mirror port:


    F.e. we see a paket here which has something to to with SIP. We also see the 0800 ethertype at byte 16 and 17 but it is at the wrong postion. Shouldnt it be at byte 12-13? I dont know where 70 34 and a0 01 do come from. They are in every packet.

    Dropboxlink to that pcap-file (about 10 MB): https://www.dropbox.com/s/2tmwfgbyui1tz1k/purviewmirror.pc.png?dl=0

    Ethertype 0800 or 8100 (for tagged) is always at byte 16 and 17 in the captures.

    SSA-G8018-0652 FW:08.62.01.0034


  • 13.  RE: purview receives netflow and mirrorN, but apps are not identified

    Posted 11-08-2016 17:44
    Where was the trace taken?


  • 14.  RE: purview receives netflow and mirrorN, but apps are not identified

    Posted 11-08-2016 19:30
    A computer with windows wireshark was directly connected to SSA ge.1.17.


  • 15.  RE: purview receives netflow and mirrorN, but apps are not identified

    Posted 11-08-2016 19:37
    I would open a GTAC ticket, it does not appear to be an obvious problem.

    https://gtacknowledge.extremenetworks.com/articles/How_To/How-to-contact-Extreme-Networks-Global-Tec...



  • 16.  RE: purview receives netflow and mirrorN, but apps are not identified

    Posted 11-09-2016 05:11
    I notice I have these two lines in my config:

    set netflow export-data enable mac
    set netflow export-data enable vlan



  • 17.  RE: purview receives netflow and mirrorN, but apps are not identified

    Posted 11-09-2016 09:24
    Since it is unlikely that policy mirroring is totally broken I sent the mirror towards another switch port.
    set mirror ports ge.1.19 2[/code]Now it worked, packets were clearly decodable. :-)

    When I configure the mirror back to ge.1.17 the problem occurs again. So the problem sticks with the switch port. Maybe it is because before trying the policy mirror config, ge.1.17 was used as target port for a GRE-Tunnel before ("tunnel mode gre l2 ge.1.17" in tun-interface definition). The GRE-tunnel mirror worked perfectly. When I started to change to policy mirror configuration I deleted the loopback and tunnel-interfaces. But maybe switch port ge.1.17 still sticks somehow in that mode. I can reboot the router at night to see if ge.1.17 gets "normal" after a reboot.



  • 18.  RE: purview receives netflow and mirrorN, but apps are not identified

    Posted 11-10-2016 06:37
    Hi,
    after a reboot the affected port returned to normal policy mirror behaviour too.


  • 19.  RE: purview receives netflow and mirrorN, but apps are not identified

    Posted 11-10-2016 15:05
    Worth Noting, on the netflow. The reason it's reporting no bi-rdirectional because the netflow config is only sending RX.|

    Should look something like "set netflow port ge.1.45 enable both"



  • 20.  RE: purview receives netflow and mirrorN, but apps are not identified

    Posted 11-10-2016 15:26
    Hi Thomas,
    in this case "netflow enable rx" is not a problem. Seeing this SSA as a border router between two networks, ge.1.45 and tg.1.4 are "outside" and the two LAGs are "inside". All of them are set to rx, so traffic getting into the net domain and flowing out of the domain gets detected. When the appliance startet to decodes the traffic properly, the "no bidirectional traffic"-message disappeared.

    Purview Deployment guide suggest setting netflow to rx on inside and outside interface f.e. to prevent double netflow detection and since policy mirroring works only ingress too. I wonder if one can combine the ingress mirrors on insinde and outside with a "netflow both" on the inside for Purview detection. hmmm...



  • 21.  RE: purview receives netflow and mirrorN, but apps are not identified

    Posted 11-09-2016 09:24
    Since the packets seemed to be encapsulated according to your packet traces, this seems plausible.