BGP routes not installed into Linux kernel routing table

terrorgen · December 2023

Hello LESbians who are proficient in BGP,

I am observing that my BIRD is not exporting any routes to the kernel.

Looks like it is receiving routes from the upstream alright:

~ terrorgen@awesomebox
❯ sudo birdc s p a upstream
BIRD 2.14 ready.
Name       Proto      Table      State  Since         Info
upstream BGP        ---        up     00:18:11.949  Established
  BGP state:          Established
    Neighbor address: 2001:db8:0:2::1
    Neighbor AS:      65000
    Local AS:         65001
    Neighbor ID:      1.2.3.4
    Local capabilities
      Multiprotocol
        AF announced: ipv6
      Route refresh
      Graceful restart
        Restart time: 120
        AF supported: ipv6
        AF preserved:
      4-octet AS numbers
      Enhanced refresh
      Long-lived graceful restart
    Neighbor capabilities
      Multiprotocol
        AF announced: ipv6
      Route refresh
      Graceful restart
      4-octet AS numbers
    Session:          external multihop AS4
    Source address:   2001:db8:3::30
    Hold timer:       75.387/90
    Keepalive timer:  21.792/30
  Channel ipv6
    State:          UP
    Table:          master6
    Preference:     100
    Input filter:   (unnamed)
    Output filter:  (unnamed)
    Routes:         195136 imported, 4 exported, 195134 preferred
    Route change stats:     received   rejected   filtered    ignored   accepted
      Import updates:         517095          0          0          0     517095
      Import withdraws:        70578          0        ---          0      70578
      Export updates:         517100     517093          3        ---          4
      Export withdraws:        70578        ---        ---        ---          0
    BGP Next hop:   2001:db8:3::30
    IGP IPv6 table: master6

But upon inspecting the individual routes, many of them are marked "!". According to the manual, that means "failure to install to kernel".

~ terrorgen@awesomebox 7s
❯ sudo birdc s r table master6
BIRD 2.14 ready.
Table master6:
::/0                 unicast [upstream 10:44:46.318 from 2001:db8:0:2::1] ! (100/?) [AS65000i]
        via 2001:db8:3::1 on enp3s0
2620:129:2000::/42   unicast [upstream 10:44:48.950 from 2001:db8:0:2::1] ! (100/?) [AS27016i]
        via 2001:db8:3::1 on enp3s0
2804:505c::/32       unicast [upstream 10:44:49.979 from 2001:db8:0:2::1] ! (100/?) [AS268384i]
        via 2001:db8:3::1 on enp3s0
2804:385c:a200::/40  unicast [upstream 10:44:49.960 from 2001:db8:0:2::1] ! (100/?) [AS262687i]
        via 2001:db8:3::1 on enp3s0
2804:d84:f000::/36   unicast [upstream 10:44:49.333 from 2001:db8:0:2::1] ! (100/?) [AS28171i]
        via 2001:db8:3::1 on enp3s0
2a04:26c0::/29       unicast [upstream 10:44:50.916 from 2001:db8:0:2::1] ! (100/?) [AS60532i]
        via 2001:db8:3::1 on enp3s0
2804:5a64::/34       unicast [upstream 10:44:50.720 from 2001:db8:0:2::1] ! (100/?) [AS268770i]
        via 2001:db8:3::1 on enp3s0
2804:5d3c:c000::/34  unicast [upstream 10:44:50.014 from 2001:db8:0:2::1] ! (100/?) [AS268959i]
        via 2001:db8:3::1 on enp3s0
2001:44b8:4040::/48  unicast [upstream 10:44:46.961 from 2001:db8:0:2::1] * (100/?) [AS7545i]
        via 2001:db8:3::1 on enp3s0
2803:bc40:8155::/48  unicast [upstream 10:44:49.180 from 2001:db8:0:2::1] ! (100/?) [AS52468i]
        via 2001:db8:3::1 on enp3s0
240a:aded::/32       unicast [upstream 10:44:48.046 from 2001:db8:0:2::1] * (100/?) [AS146215i]
        via 2001:db8:3::1 on enp3s0
2402:3a80:19ec::/46  unicast [upstream 10:44:47.367 from 2001:db8:0:2::1] * (100/?) [AS38266i]
        via 2001:db8:3::1 on enp3s0
2a00:1d58:f812::/48  unicast [upstream 10:44:50.213 from 2001:db8:0:2::1] ! (100/?) [AS47524?]
        via 2001:db8:3::1 on enp3s0
2620:12f:f000::/44   unicast [upstream 10:44:49.021 from 2001:db8:0:2::1] ! (100/?) [AS43?]
        via 2001:db8:3::1 on enp3s0
2800:bf0:2206::/48   unicast [upstream 10:44:48.906 from 2001:db8:0:2::1] ! (100/?) [AS52257i]
        via 2001:db8:3::1 on enp3s0
2001:4430:c20e::/47  unicast [upstream 10:44:46.931 from 2001:db8:0:2::1] * (100/?) [AS17853i]
        via 2001:db8:3::1 on enp3s0
...

Looking at systemd-journald this is the message being generated (restarted BIRD just to get the full message):

~ terrorgen@awesomebox 11s
❯ sudo systemctl restart bird2; sudo journalctl -fu bird2
Dec 04 10:50:36 awesomebox bird[773]: Shutdown completed
Dec 04 10:50:36 awesomebox systemd[1]: bird2.service: Deactivated successfully.
Dec 04 10:50:36 awesomebox systemd[1]: Stopped BIRD Internet Routing Daemon.
Dec 04 10:50:36 awesomebox systemd[1]: bird2.service: Consumed 10.171s CPU time, received 8.5M IP traffic, sent 17.9M IP traffic.
Dec 04 10:50:36 awesomebox systemd[1]: Starting BIRD Internet Routing Daemon...
Dec 04 10:50:36 awesomebox systemd[1]: Started BIRD Internet Routing Daemon.
Dec 04 10:50:36 awesomebox bird[1047]: Started
Dec 04 10:50:38 awesomebox bird[1047]: Netlink: File exists
Dec 04 10:50:38 awesomebox bird[1047]: Kernel dropped some netlink messages, will resync on next scan.
Dec 04 10:50:39 awesomebox bird[1047]: Kernel dropped some netlink messages, will resync on next scan.
Dec 04 10:50:39 awesomebox bird[1047]: Kernel dropped some netlink messages, will resync on next scan.
Dec 04 10:50:39 awesomebox bird[1047]: Kernel dropped some netlink messages, will resync on next scan.
(multiple identical messages)
Dec 04 10:50:41 awesomebox bird[1047]: Netlink: No route to host
Dec 04 10:50:41 awesomebox bird[1047]: ...
Dec 04 10:50:42 awesomebox bird[1047]: Netlink: No route to host
Dec 04 10:50:42 awesomebox bird[1047]: ...
Dec 04 10:50:43 awesomebox bird[1047]: Netlink: No route to host
Dec 04 10:50:43 awesomebox bird[1047]: ...
Dec 04 10:50:44 awesomebox bird[1047]: Netlink: No route to host
Dec 04 10:50:44 awesomebox bird[1047]: ...
Dec 04 10:50:45 awesomebox bird[1047]: Netlink: No route to host
Dec 04 10:50:45 awesomebox bird[1047]: ...
Dec 04 10:50:46 awesomebox bird[1047]: Netlink: No route to host
Dec 04 10:50:46 awesomebox bird[1047]: ...
Dec 04 10:50:47 awesomebox bird[1047]: Netlink: No route to host
Dec 04 10:50:47 awesomebox bird[1047]: ...

Wonder if you have any pointers?

Note: 2001:db8:3::30 is my VPS address, upstream is using multihop. Immediate nexthop router is 2001:db8:3::1 and BGP router is at 2001:db8:0:2::1.

GeekWanderer · December 2023

How does the kernel section of your bird.conf look?

terrorgen · December 2023

@GeekWanderer said:
How does the kernel section of your bird.conf look?

quite boiler plate...

protocol kernel {
        metric 1024;
        ipv6 {
                import none;
                export where source ~ [RTS_BGP]; 
        };
        merge paths yes;
}

Neoon · December 2023

bird has multiple tables, you have to explicitly tell it to do that.

I am not an expert in bird though.
Maybe @FHR can help.

terrorgen · December 2023

there is a default master4 and master6 table. If you don't specify the table in the config the default tables are used.

That being said, this is not the first nor the only BGP session I have. All sessions are written with the snippet above and there isn't a problem.

FHR · December 2023

Is this actually a valid route installed in the kernel (aka visible in ip -6 r)?
2001:db8:3::1 on enp3s0 If netlink is complaining about no route to host while trying to install a route it sort of points towards that not being the case; i.e. Bird is trying to insert a route that is not resolveable by the kernel.

terrorgen · December 2023

that is the statically configured default route:
default via 2001:db8:3::1 dev enp3s0 proto static metric 1024 pref medium

c1vhosting · December 2023

Pls share your entire bird config

terrorgen · December 2023

router id 4.3.2.1;
log syslog all;

function is_self_net() -> bool {
        return net ~ 2001:db8:abcd::/48;
}

protocol device {
        scan time 10;
}

protocol static sss {
        ipv6;
        route 2001:db8:abcd::/48 unreachable;
}

protocol kernel {
        metric 1024;
        ipv6 {
                import none;
                export where source ~ [RTS_BGP]; 
        };
        merge paths yes;
}

template bgp peers
{
        local as 65001;
        ipv6 {
                import filter {
                        if (is_self_net()) then reject;
                        accept;
                };
                export where proto = "sss";
        };
        graceful restart on;
}

protocol bgp upstream from peers
{
        multihop;
        source address 2001:db8:3::30;
        neighbor 2001:db8:0:8::1 as 65000;
}

c1vhosting · December 2023

protocol kernel {
ipv6 {
import all;
export all; };
}

terrorgen · December 2023

@c1vhosting said:
protocol kernel {
ipv6 {
import all;
export all; };
}

Why?
It is potentially dangerous to just import and export all. Plus, it doesn't make any difference to the problem anyways.

dwight · December 2023

@terrorgen said:
Why?
It is potentially dangerous to just import and export all. Plus, it doesn't make any difference to the problem anyways.

kernel export all isn't that dangerous. Anyway, I guess this article will help you: https://blog.fhrnet.eu/2018/08/15/bird-unreachable-routes-on-multihop-bgp-sessions/

terrorgen · December 2023

@dwight said:

show previous quotes

@terrorgen said:
Why?
It is potentially dangerous to just import and export all. Plus, it doesn't make any difference to the problem anyways.

kernel export all isn't that dangerous. Anyway, I guess this article will help you: https://blog.fhrnet.eu/2018/08/15/bird-unreachable-routes-on-multihop-bgp-sessions/

Thanks but unfortunately, that doesn't describe my problem though. I can reach the multi hop router fine.

GeekWanderer · December 2023

What did you determine with this? Find a resolution?

terrorgen · December 2023

Somehow I have to set the next hop router's address to "onlink" for the routes to install, which is weird. The next hop router is reachable anyways without setting this.

AuroraZero · December 2023

@terrorgen said:
Somehow I have to set the next hop router's address to "onlink" for the routes to install, which is weird. The next hop router is reachable anyways without setting this.

Lmfao wth man?

terrorgen · December 2023

@AuroraZero said:

show previous quotes

@terrorgen said:
Somehow I have to set the next hop router's address to "onlink" for the routes to install, which is weird. The next hop router is reachable anyways without setting this.

Lmfao wth man?

IKR?

AuroraZero · December 2023

@terrorgen said:

show previous quotes

@AuroraZero said:

@terrorgen said:
Somehow I have to set the next hop router's address to "onlink" for the routes to install, which is weird. The next hop router is reachable anyways without setting this.

Lmfao wth man?

IKR?

How did that even work?

terrorgen · December 2023

@AuroraZero said:

show previous quotes

@terrorgen said:

@AuroraZero said:

@terrorgen said:
Somehow I have to set the next hop router's address to "onlink" for the routes to install, which is weird. The next hop router is reachable anyways without setting this.

Lmfao wth man?

IKR?

How did that even work?

Might need someone who truly understands the Linux kernel network stack to explain it to us.

AuroraZero · December 2023

@terrorgen said:

show previous quotes

@AuroraZero said:

@terrorgen said:

@AuroraZero said:

@terrorgen said:
Somehow I have to set the next hop router's address to "onlink" for the routes to install, which is weird. The next hop router is reachable anyways without setting this.

Lmfao wth man?

IKR?

How did that even work?

Might need someone who truly understands the Linux kernel network stack to explain it to us.

What I am saying is it should not have even partially worked.

Someone here should know I hope because it may not even be replacable.

terrorgen · December 2023

@AuroraZero said:

show previous quotes

@terrorgen said:

@AuroraZero said:

@terrorgen said:

@AuroraZero said:

@terrorgen said:
Somehow I have to set the next hop router's address to "onlink" for the routes to install, which is weird. The next hop router is reachable anyways without setting this.

Lmfao wth man?

IKR?

How did that even work?

Might need someone who truly understands the Linux kernel network stack to explain it to us.

What I am saying is it should not have even partially worked.

Someone here should know I hope because it may not even be replacable.

Oh, yes, very true. Why does the kernel choose to accept some prefixes and deny some other, when their nexthops are the same?

AuroraZero · December 2023

@terrorgen said:

show previous quotes

@AuroraZero said:

@terrorgen said:

@AuroraZero said:

@terrorgen said:

@AuroraZero said:

@terrorgen said:
Somehow I have to set the next hop router's address to "onlink" for the routes to install, which is weird. The next hop router is reachable anyways without setting this.

Lmfao wth man?

IKR?

How did that even work?

Might need someone who truly understands the Linux kernel network stack to explain it to us.

What I am saying is it should not have even partially worked.

Someone here should know I hope because it may not even be replacable.

Oh, yes, very true. Why does the kernel choose to accept some prefixes and deny some other, when their nexthops are the same?

No idea man and I have not heard of one acting that strange before. Is it a stock kernel?

BGP routes not installed into Linux kernel routing table

Comments