Full Rack Colocation Newbie: Advice Please + Networking?

lentro

I've been relying on single server colo so far where DCs handle PDUs and networking, but by this summer (yes I shall be a summer host ), it looks like I will be at the scale to expand to full racks. I've received some great quotes so I just want to wrap my head around everything and know exactly what I'll need to do.

I'm have a developer background so this is a bit new to me.

I need networking equipment that enables me to do:

• 1 gbps ideally with 10 gbps uplink
• Managed so QOS, rate limiting, etc
• Can announce IPs
• Can have the DC run at least 2 cross connects to it (so a normal rackmount switch)

So far, I've come across this:

• Cisco 48 port 10 gbps switch w/10 gbps SFP+

Any thoughts/advice?

In general, what should I know about full rack colo?

qps

Unless you have to have a Cisco, look into a Juniper EX4300-48T plus EX-UM-4X4SFP. Really good bang for the buck given current eBay pricing.

There are two versions depending on how you plan to mount it in your rack. If you plan to mount it at the front of the cabinet, EX4300-48T is the correct part number. If you plan to mount it at the back of the cabinet (this is most common in data center installs), you want the EX4300-48T-AFI.

As you probably know, you will need to maintain your own spares. You will probably want to buy a rackmount shelf that you can store stuff in unless the data center has a storage locker that you can rent.

Not_Oles

@lentro said: I've received some great quotes

Want to be at HE? How about a free 42U full cabinet, A&B (primary and redundant) 20 amp 208 volt power, 1 Gbps on gige Internet connection? Maybe you kindly might let me put a 1U hobby server in your cabinet?

lentro

@Not_Oles said:

@lentro said: I've received some great quotes

Want to be at HE? How about a free 42U full cabinet, A&B (primary and redundant) 20 amp 208 volt power, 1 Gbps on gige Internet connection? Maybe you kindly might let me put a 1U hobby server in your cabinet?

Wait what? Can you elaborate? I don't fully understand.

So these are for GPU servers, some quotes are for 30a 208v (5 kW after 80% rule) and some are 4x 30a 208v (20 kW after 80% rule). But hey, I might be able to share some resources for you to play around with

HE's quotes have been quite expensive tbh for high power density racks. Makes sense given they're in California and not say Quincy Washington

alwyzon

@lentro said:

• 1 gbps ideally with 10 gbps uplink
• Managed so QOS, rate limiting, etc
• Can announce IPs

Yep, sounds like you are looking for a Juniper EX3400-24T or EX3400-48T. The first one (the 24 port one) starts at about ~1100€ new (plus ~800 € for the license required to use BGP). They do even have two QFSP+ port (ideal for creating a ring of multiple switches in different racks) and are likely more energy-efficient than any old, used equipment you might find on eBay.

— Michael

Not_Oles

@lentro Through the years, just anecdotally and without any measurements, it's always made me deliciously happy when traceroute tells me I am routed via HE. The connections via HE somehow always seem faster and cleaner to me than anything else. Also, I briefly had an HE VPS and I have tried HE's DNS service. Top notch.

There is a many years standing email thread between me and Mike Leber. I wanted a way to get a server inside HE, but for cheap a Low End price. In 2019 Mike suggested that I find HE a customer with its own ASN and address space and knowledge of how to run BGP. Mike said HE would give the customer a free cabinet if the customer installs a real router and lists themselves in peeringdb as being present in HE's datacenter. Mike suggested that the customer might be willing to let me put two servers in the free rack I arranged for them.

If you or anyone else is seriously interested, we maybe can talk more with Mike.

lentro

@Not_Oles said: HE

PM'd! Really interested although not sure about the sustainability of such a loss leader for them, so I'd want to work out something where I at least pay for HE's costs so they, at a minimum, break even.

martijnk

There are lots of things to consider. I did the same last year

First of all, you need redundancy. So ideally you need A+B power feeds but you also need stackable switches so that if one goes offline the other one takes over.

The other thing is storage. Are you going for networked storage (e.g. CEPH) or local. If you pick networked, 1 gbps is not enough and you should ideally run 10 gbps switching at a minimum. I myself bought 40 GBit QSFP+ Dell switches (u can pick them up for around $1000) and use Mellanox network interfaces in the nodes. So I have 80 gbit (2 x 40 GBit LACP) everywhere.

For public network you can use another switch but remember to buy ones that support stacking. For public I have a couple of N2024s and my provider is providing me with LACP uplinks.

Btw I have DELL S6010-on switches. https://www.ebay.com/itm/174721923739?epid=2158735074&hash=item28ae3cfa9b:g:jvQAAOSwDF5gEIaa

hostEONS

@lentro if you don't have experience with managing your own switch/router, I would recommend to contact your DC to provide a leased and managed switch/router

I should cost just $100-$200 per month extra, but you won't need to bother about managing your router and also need not worry about keep spares

lentro

@martijnk thanks! Very nice info. No network storage for now so 1 gbps should be enough but will definitely consider 40gbps for future-proofing! Thanks!

CamoYoshi

@lentro said:
HE's quotes have been quite expensive tbh for high power density racks. Makes sense given they're in California and not say Quincy Washington

The Quincy, WA datacenter is one of a few my company I work for uses. It's excellent and super reliable for us and we have a few rows of racks there. It's been great as a load balanced/DR site to go along with some of our other DCs.

martijnk

@hostEONS said:
@lentro if you don't have experience with managing your own switch/router, I would recommend to contact your DC to provide a leased and managed switch/router

I should cost just $100-$200 per month extra, but you won't need to bother about managing your router and also need not worry about keep spares

It pays of to learn, it's not rocket science. Everything you can do yourself will save you money. If you don't even want to manage your own switches then just go with dedicated servers.

It takes some time to figure out but once up and running there is hardly anything to do and you will save that $100-$200 every single month.

PureVoltage

Personally I would look at going with maybe a Ex4200 Juniper.

Depending on the usage for your switch learning is the best option, however if you can get one configured for you depending on the service you are planning on offering.
If it's not needing a bunch of Vlans setup etc a dummy switch is easy to setup once and it's good for a long time.

With colocation there are a lot of things to be aware of, power, bandwidth, remote hands etc.
The more you know about the places the better.

Some charge $75 just to reboot a server, others do this free.
WA state Quincy is the cheapest in the US if you need lots of power. However bandwidth options are more limited there.