Full Rack Colocation Newbie: Advice Please + Networking?

lentrolentro Hosting Provider

I've been relying on single server colo so far where DCs handle PDUs and networking, but by this summer (yes I shall be a summer host :joy: ), it looks like I will be at the scale to expand to full racks. I've received some great quotes so I just want to wrap my head around everything and know exactly what I'll need to do.

I'm have a developer background so this is a bit new to me.

I need networking equipment that enables me to do:

  • 1 gbps ideally with 10 gbps uplink
  • Managed so QOS, rate limiting, etc
  • Can announce IPs
  • Can have the DC run at least 2 cross connects to it (so a normal rackmount switch)

So far, I've come across this:

Any thoughts/advice?

In general, what should I know about full rack colo?

Comments

  • qpsqps Hosting ProviderOG

    Unless you have to have a Cisco, look into a Juniper EX4300-48T plus EX-UM-4X4SFP. Really good bang for the buck given current eBay pricing.

    There are two versions depending on how you plan to mount it in your rack. If you plan to mount it at the front of the cabinet, EX4300-48T is the correct part number. If you plan to mount it at the back of the cabinet (this is most common in data center installs), you want the EX4300-48T-AFI.

    As you probably know, you will need to maintain your own spares. You will probably want to buy a rackmount shelf that you can store stuff in unless the data center has a storage locker that you can rent.

    Thanked by (4)ehab lentro ialexpw Wolveix
  • Not_OlesNot_Oles Hosting ProviderContent Writer

    @lentro said: I've received some great quotes

    Want to be at HE? How about a free 42U full cabinet, A&B (primary and redundant) 20 amp 208 volt power, 1 Gbps on gige Internet connection? Maybe you kindly might let me put a 1U hobby server in your cabinet? :)

    I hope everyone gets the servers they want!

  • lentrolentro Hosting Provider

    @Not_Oles said:

    @lentro said: I've received some great quotes

    Want to be at HE? How about a free 42U full cabinet, A&B (primary and redundant) 20 amp 208 volt power, 1 Gbps on gige Internet connection? Maybe you kindly might let me put a 1U hobby server in your cabinet? :)

    Wait what? Can you elaborate? I don't fully understand.

    So these are for GPU servers, some quotes are for 30a 208v (5 kW after 80% rule) and some are 4x 30a 208v (20 kW after 80% rule). But hey, I might be able to share some resources for you to play around with :)

    HE's quotes have been quite expensive tbh for high power density racks. Makes sense given they're in California and not say Quincy Washington :joy:

  • alwyzonalwyzon Hosting Provider

    @lentro said:

    • 1 gbps ideally with 10 gbps uplink
    • Managed so QOS, rate limiting, etc
    • Can announce IPs

    Yep, sounds like you are looking for a Juniper EX3400-24T or EX3400-48T. The first one (the 24 port one) starts at about ~1100€ new (plus ~800 € for the license required to use BGP). They do even have two QFSP+ port (ideal for creating a ring of multiple switches in different racks) and are likely more energy-efficient than any old, used equipment you might find on eBay.

    — Michael

    Thanked by (1)lentro

    Alwyzon - Virtual Servers in Austria starting at 4,49 €/month (excl. VAT)

  • Not_OlesNot_Oles Hosting ProviderContent Writer

    @lentro Through the years, just anecdotally and without any measurements, it's always made me deliciously happy when traceroute tells me I am routed via HE. The connections via HE somehow always seem faster and cleaner to me than anything else. Also, I briefly had an HE VPS and I have tried HE's DNS service. Top notch.

    There is a many years standing email thread between me and Mike Leber. I wanted a way to get a server inside HE, but for cheap a Low End price. In 2019 Mike suggested that I find HE a customer with its own ASN and address space and knowledge of how to run BGP. Mike said HE would give the customer a free cabinet if the customer installs a real router and lists themselves in peeringdb as being present in HE's datacenter. Mike suggested that the customer might be willing to let me put two servers in the free rack I arranged for them. :)

    If you or anyone else is seriously interested, we maybe can talk more with Mike.

    Thanked by (1)lentro

    I hope everyone gets the servers they want!

  • lentrolentro Hosting Provider

    PM'd! Really interested although not sure about the sustainability of such a loss leader for them, so I'd want to work out something where I at least pay for HE's costs so they, at a minimum, break even.

  • edited April 2021

    There are lots of things to consider. I did the same last year ;)

    First of all, you need redundancy. So ideally you need A+B power feeds but you also need stackable switches so that if one goes offline the other one takes over.

    The other thing is storage. Are you going for networked storage (e.g. CEPH) or local. If you pick networked, 1 gbps is not enough and you should ideally run 10 gbps switching at a minimum. I myself bought 40 GBit QSFP+ Dell switches (u can pick them up for around $1000) and use Mellanox network interfaces in the nodes. So I have 80 gbit (2 x 40 GBit LACP) everywhere.

    For public network you can use another switch but remember to buy ones that support stacking. For public I have a couple of N2024s and my provider is providing me with LACP uplinks.

    Btw I have DELL S6010-on switches. https://www.ebay.com/itm/174721923739?epid=2158735074&hash=item28ae3cfa9b:g:jvQAAOSwDF5gEIaa

    Thanked by (1)lentro
  • hostEONShostEONS Hosting ProviderOG

    @lentro if you don't have experience with managing your own switch/router, I would recommend to contact your DC to provide a leased and managed switch/router

    I should cost just $100-$200 per month extra, but you won't need to bother about managing your router and also need not worry about keep spares

    Thanked by (1)lentro

    hostEONS.com - 9 Locations for VPS (US and EU).. Free Blesta License, Free Windows 2019 License and more. Shared DA Hosting. AS142036 .. Ryzen 7950X Based VDS in SLC, LA and Dallas

  • lentrolentro Hosting Provider

    @martijnk thanks! Very nice info. No network storage for now so 1 gbps should be enough but will definitely consider 40gbps for future-proofing! Thanks!

  • @lentro said:
    HE's quotes have been quite expensive tbh for high power density racks. Makes sense given they're in California and not say Quincy Washington :joy:

    The Quincy, WA datacenter is one of a few my company I work for uses. It's excellent and super reliable for us and we have a few rows of racks there. It's been great as a load balanced/DR site to go along with some of our other DCs.

    Cheap dedis are my drug, and I'm too far gone to turn back.

  • @hostEONS said:
    @lentro if you don't have experience with managing your own switch/router, I would recommend to contact your DC to provide a leased and managed switch/router

    I should cost just $100-$200 per month extra, but you won't need to bother about managing your router and also need not worry about keep spares

    It pays of to learn, it's not rocket science. Everything you can do yourself will save you money. If you don't even want to manage your own switches then just go with dedicated servers.

    It takes some time to figure out but once up and running there is hardly anything to do and you will save that $100-$200 every single month.

    Thanked by (3)yoursunny lentro CamoYoshi
  • PureVoltagePureVoltage Hosting ProviderOG

    Personally I would look at going with maybe a Ex4200 Juniper.

    Depending on the usage for your switch learning is the best option, however if you can get one configured for you depending on the service you are planning on offering.
    If it's not needing a bunch of Vlans setup etc a dummy switch is easy to setup once and it's good for a long time.

    With colocation there are a lot of things to be aware of, power, bandwidth, remote hands etc.
    The more you know about the places the better.

    Some charge $75 just to reboot a server, others do this free.
    WA state Quincy is the cheapest in the US if you need lots of power. However bandwidth options are more limited there.

    Thanked by (1)lentro

    PureVoltage - Custom Dedicated Servers Dual E5-2680v3 64gb ram 1TB nvme 100TB/10g $145
    New York Colocation - Amazing pricing 1U-48U+

Sign In or Register to comment.