Dear customers, partners, friends, nerds and competing internet exchanges! Today, we are proud to announce something completely out-of-the-box. As most of you know, internet exchanges publish quite some valuable information about their operations; think about traffic statistics and connected members. This time we have come up with something exciting!

Background

When we founded ERA-IX back in 2022, one of the major goals was to build a cost-effective alternative to the rusty giants, not only the customer side, but also our infrastructure. We used efficient switching equipment and servers to make this happen, and we are quite proud about this because as of today, we only consume around 3 kW, while pushing more than ~1.2 terabits of traffic on the exchange daily.

Something new

So when we are running very efficiently, why not share it publicly? Today is that day, and we now publish our power use on our Statistics page. We now show the "Total Power", this is all the power our platform and supporting devices utilize. On top of that we show "Power per Member", which is the average power use per connected member, and finally the "Power per Gigabit".

For those curious, let's go through all the major parts that utilize power in our platform:

• Arista core switching equipment, we recently upgraded our platform to new Arista R3 series for 400G port support.
• Supporting switches, based on more power-efficient chips mainly for interconnectivity and OOB access.
• Teltonika OOB routers (made in EU), in case of emergencies.
• Two Route Servers Amsterdam.
• Two Route Servers Frankfurt.
• Transceivers; 10G-LR, 40G-LR, 40G-PLR4, 100G-LR4, 100G-IR4, 100G-LR1, 100G BiDis, 400G LR4.
• Infrastructure cluster for all supporting services like website, portal, graphing, and backend systems.
• Backup servers in both metros, in different datacenters.

As we've connected around 110 100G interfaces towards members, if we assume each 100G-LR4 transceiver consumes around 3.5 watts of power, 380W (>10%) of our power consumption can already be attributed to physical interconnections towards members.

Note that these metrics are all based on real data, polled from our equipment in real time. We have seen some very optimistic figures posted online, typically based on chipset specifications, while in the real world ideal conditions are rarely encountered and efficiency numbers (Watts per Gbit) will be in a different magnitude from the figure mentioned.

Total Power:

Power per Member:

Power per Gigabit:

The chase for the most efficient packets

Our 7 day average Joules per Gbit is 3,178; (= Watts per Gbit), our website actually uses median metrics instead of average, which is slightly less optimistic, but more in the 'middle' of our operating conditions.

Average: Traffic peaks would greatly reduce the Joules per Gbit as large peaks, which cases negligible additional power use push down the average a lot. Median: Takes the center sample of our traffic volume and power usage, which is closer to our actual operating conditions most of the time.

While efficiency is very high on the list of our priorities, there are also some requirements our platform needs to fulfill, which is the reason we don't run Tomahawk devices everywhere.

Some other steps we're taking to minimize power usage;

• Utilizing DAC cables for short interconnects within our racks. While harder to manage physically, they are able to save quite a bit of power compared to optical transceivers, typically operating at a few watts a piece;
• Ensuring our racks are always properly blind plated and running our switches port-side intake, so we need less fan speed to keep our transceivers nice and cool;
• Putting transceivers that we are not actively using in 'LP mode' instead of leaving the interfaces on or shutting the interfaces, this easily saves another continuous 70 watts of power (based on stock).

To show the impressive gains of LP mode, 100G-SR4 transceiver in low-power mode:

100G-SR4 in standard mode:

For 100G member interconnection on QSFP28 switches, we've made the (rather backwards) discovery that at this moment 100G-LR4 is probably a more efficient media than 100G-LR(1) We've theorized this being due to the need for an additional processing step to go from four to just one lane, and the other way around. With 112Gbps serdes, this processing step will no longer be necessary and single-laser 100G will be much more efficient.

As you can see in the power graphs, we started deploying new R3 switching equipment in October 2024, mainly to offer more features and 400G ports. This eventually leads to more power use, but this is caused by the new platform, as these switches are much more capable and are fully VoQ. This ensures fair queuing discipline during bandwidth contention on members ports. Furthermore, due to the virtually lossless operation of these switches, we're able to maintain a exact 100% unicast packet delivery rate monitored down to the second. Eventually, the introduction of 400G ports allows us to exchange traffic even more efficiently as there will also be less components to maintain, fail and power for the highest bandwidth requirements.

How do we measure?

For standard monitoring of our network equipment, we use the open-source LibreNMS. All our switches support reading input voltages and current at the AC input of the power supply meaning we can get an accurate figure of how much power a device is using. While we can only calculate apparent power, we assume unity power factor, if the power factor of our power supplies is not unity, this would cause our real consumption to below the given figure. We've measured the power factor of our PSUs to be very close to unity.

What does the future hold?

More bits, less power! Our efficient core is also the reason we're able to maintain sharp prices in an environment of inflation, while bringing ERA-IX to more and more datacenters.

This article was co-written with my colleague Elias.

About Eranium

We are a young initiative born and raised in the Achterhoek, a region east of the Netherlands. Fully bootstrapped, meaning we are privately owned, debt-free, and investor-free. With our direct and down-to-earth approach we captured the hearts of fellow network enthusiasts and together with our partners and customers, we strive to be in the top of the connectivity providers in Europe.

---