Articles & Blogs
Data center interconnect (DCI) is private network connectivity between multiple data center facilities that lets you treat geographically separated infrastructure as a unified environment. Instead of routing traffic between sites over the public internet, DCI uses dedicated circuits that provide predictable latency, better security, and guaranteed bandwidth. This enables disaster recovery, live workload migration, data replication, and distributed applications that span multiple locations.
Most organizations with serious infrastructure requirements operate in multiple data centers. You need geographic diversity for disaster recovery, capacity in different markets to serve regional users, or the ability to scale beyond what a single facility provides. The question isn’t whether you’ll need multi-site infrastructure, but how you’ll connect those sites so they work together effectively.
Internet connectivity between sites works for some purposes, but it introduces latency variability, security concerns, and unreliable performance that make it unsuitable for tightly coupled applications. DCI eliminates these issues by providing private, dedicated connectivity that performs more like a local network than a wide area connection.
What is Data Center Interconnect (DCI)?
Data center interconnect refers to dedicated network connections between data center facilities that carry traffic between your infrastructure at each location. These connections can use various technologies – dark fiber, wavelength services, Ethernet circuits, MPLS networks – but they all provide private connectivity that doesn’t traverse the public internet.
The key characteristic is that DCI connects your infrastructure to your infrastructure. This differs from internet connectivity, where you connect to the broader internet and route traffic to destinations that might include your other sites. DCI creates direct paths between specific facilities you control.
Why Organizations Deploy Data Center Interconnect
Running infrastructure across multiple locations creates the need to move data between sites. Database replication for disaster recovery, application traffic flowing between distributed components, centralized backup of data from multiple locations, workload migration between facilities – all these scenarios work better with dedicated connectivity than internet paths.
Performance matters for these use cases. Database replication needs consistent low latency to maintain synchronization. Application traffic between sites requires predictable bandwidth to deliver a good user experience. Backup operations benefit from guaranteed capacity that doesn’t compete with internet traffic.
Security becomes simpler when traffic flows over private circuits rather than the public internet. You don’t need to worry about internet-based attacks intercepting data in transit between your facilities. The circuit only carries your traffic, providing inherent isolation.
DCI Network Topologies
Point-to-point DCI connects two facilities with a single dedicated circuit. This works well when you operate two primary sites that need high-bandwidth, low-latency connectivity between them. The circuit runs directly from one facility to the other, providing the simplest and often lowest-latency option.
Hub-and-spoke topologies connect multiple facilities to a central hub location. Remote sites connect to the hub, and traffic between remote sites routes through the hub. This reduces the number of circuits needed compared to full mesh connectivity but introduces an extra hop for site-to-site traffic.
Full mesh topologies provide direct connectivity between every facility. In a four-site deployment, this means six circuits connecting each possible site pair. Full mesh provides the best performance because traffic always takes the direct path, but circuit costs scale quickly as you add sites.
Hybrid topologies mix these approaches. Critical sites might have direct connections, while less critical sites connect through hub-and-spoke arrangements. This balances performance requirements with cost constraints.
Data Center Interconnect vs. Standard Internet Connectivity
Understanding the differences between DCI and internet connectivity helps clarify when the additional cost of dedicated circuits makes sense.
Performance and Latency
Internet paths route traffic through multiple carriers and exchange points, with routing decisions made dynamically based on current conditions. This creates latency variability – your traffic might take different paths at different times, with latency varying by 20-50 percent or more between measurements.
DCI provides consistent latency because traffic always follows the same physical path. A circuit between facilities delivers nearly identical latency measurements hour to hour and day to day. This consistency matters for applications sensitive to latency spikes.
The absolute latency might not differ dramatically – a 1,000-mile internet path and 1,000-mile DCI circuit both face the same physical speed-of-light limitations. But the predictability differs substantially. Internet: 25-40 milliseconds with high variability. DCI: 28-32 milliseconds with minimal variability.
Bandwidth and Throughput
Internet connections provide “best effort” bandwidth. You might pay for a 1 Gbps circuit, but actual throughput varies based on congestion at peering points, routing decisions, and overall internet conditions. During peak hours, available bandwidth often drops significantly below nominal capacity.
DCI circuits provide guaranteed bandwidth. A 10 Gbps circuit delivers 10 Gbps consistently. This makes capacity planning straightforward and ensures applications get the bandwidth they need regardless of broader network conditions.
Packet loss on internet connections typically runs 0.1-1 percent under normal conditions, but can spike higher during congestion. DCI circuits commonly deliver packet loss under 0.001 percent because they don’t face the congestion issues that affect internet paths.
Security Considerations
Internet traffic faces exposure to various attacks – man-in-the-middle attacks, traffic interception, DDoS attacks targeting the connection. You need encryption and security controls to protect data traversing the internet between sites.
DCI provides inherent security through isolation. The circuit carries only your traffic, and no one else can access it. You still might implement encryption for defense in depth, but the private nature of DCI circuits eliminates most internet-based attack vectors.
Traffic on DCI circuits isn’t visible to Internet monitoring or traffic analysis. This matters for applications where even metadata about communication patterns between sites is sensitive.
Cost Structure
Internet connectivity costs less per megabit than dedicated circuits, particularly for lower bandwidth requirements.
The cost crossover happens as bandwidth requirements increase. At 10 Gbps and beyond, DCI circuits often become more cost-effective than internet circuits delivering equivalent performance. For very high bandwidth requirements (40-100 Gbps), DCI clearly wins on per-megabit costs.
Factor in the value of consistent performance and security when comparing costs. An internet connection that costs half as much but can’t deliver required performance doesn’t save money if it prevents applications from working correctly.
Comparison Table
| Factor | Internet Connectivity | Data Center Interconnect |
| Latency | Variable (20-50% swings) | Consistent, predictable |
| Bandwidth | Best effort, varies with congestion | Guaranteed, constant |
| Packet Loss | 0.1-1% typical | <0.001% typical |
| Security | Exposed to internet threats | Private, isolated |
| Setup Time | Days to weeks | Weeks to months |
| Cost (10G+) | Higher per-megabit | Lower per-megabit |
Layer 2 and Layer 3 DCI Options
DCI circuits can operate at different network layers, with Layer 2 and Layer 3 being the most common options. Understanding these differences helps you choose the right approach for your use cases.
Layer 2 DCI Explained
Layer 2 DCI extends your local network across geographic distance. The circuit acts like a very long network cable connecting switches in different facilities. This creates a single broadcast domain spanning multiple locations – the facilities function as one large local network from a Layer 2 perspective.
The main advantage is simplicity for applications. Servers at different sites appear to be on the same local network. IP addressing stays simple because you use the same subnet across sites. Virtual machine migration works smoothly because the network environment looks identical at both locations.
Layer 2 DCI enables active-active application architectures where workloads run simultaneously at multiple sites with seamless failover. When one site has issues, workloads at the other site continue operating without IP address changes or DNS updates.
Limitations exist – spanning tree protocols that prevent Layer 2 loops can reduce available bandwidth. Broadcast traffic propagates across all sites, which can cause issues with chatty protocols. The extended failure domain means a Layer 2 problem at one site can impact all connected sites.
Layer 3 DCI Explained
Layer 3 DCI treats each facility as a separate network. Circuits carry routed IP traffic between sites rather than extending Layer 2 domains. Each site has its own IP address space, and routing protocols determine how traffic flows between locations.
This approach provides better isolation between sites. Network issues at one location don’t propagate to others through Layer 2 mechanisms. Routing protocols can use multiple paths and load balance traffic across links, providing better bandwidth utilization than spanning tree algorithms.
Layer 3 requires a more sophisticated application design. Applications need to handle failover between sites when IP addresses change. Load balancers, DNS, or application-layer failover mechanisms manage traffic distribution across locations.
Most enterprise networks use Layer 3 DCI because it scales better and provides clearer separation between sites. Layer 2 DCI makes sense for specific use cases requiring that functionality, but Layer 3 works for a broader range of scenarios.
Choosing Between Layer 2 and Layer 3
Use Layer 2 DCI when:
- You need live virtual machine migration between sites
- Applications require servers at different sites to be on the same subnet
- You’re implementing active-active architectures with automatic failover
- Your team has limited routing expertise and prefers simpler Layer 2 approaches
Use Layer 3 DCI when:
- You need to scale beyond two or three interconnected sites
- Network isolation between sites is important
- You want to use multiple circuits with load balancing
- Your applications handle multi-site deployment with IP address differences
Most organizations end up with Layer 3 DCI because it provides better scalability and isolation. The additional complexity of routing is manageable with modern tools and is worth the operational benefits.
Dark Fiber and Lit Services for DCI
Dark Fiber Services
Dark fiber means the provider gives you physical fiber strands between locations and you supply all the equipment to transmit light over those fibers. You install optical transceivers, multiplexers, and any other equipment needed to create working circuits.
The advantage is control and flexibility. You decide the transmission technology, protocol, and capacity. Want to start with 10 Gbps and later upgrade to 100 Gbps? Just change the equipment at the endpoints rather than ordering a new circuit from the provider.
Costs favor dark fiber at high bandwidth. Once you’ve paid for the fiber installation (or leased existing fiber), the ongoing monthly cost doesn’t vary much based on how much capacity you light. You can potentially run multiple wavelengths over the same fiber pair, massively increasing capacity without proportional cost increases.
The downsides include technical complexity and capital expense. You need staff with optical networking expertise or contractors who can design and maintain these systems. You buy expensive transmission equipment upfront rather than renting it from providers.
Dark fiber works well for high-bandwidth requirements between facilities you control long-term. A company connecting two large data centers that they’ll operate for years can justify the investment. Short-term needs, or lower bandwidth requirements, often work better with lit services.
Lit Services (Wavelength and Ethernet)
Lit services deliver ready-to-use circuits without requiring you to manage optical transmission. The provider handles all the physical layer complexity and gives you a handoff at the agreed capacity – 1 Gbps, 10 Gbps, 100 Gbps.
Wavelength services provide a dedicated optical wavelength between locations. You get Layer 1 connectivity at whatever speed the wavelength supports. The provider manages the optical gear; you just plug in routers or switches at endpoints.
Ethernet services deliver Layer 2 connectivity with standard Ethernet handoffs. These services often support advanced features like quality-of-service controls, VLAN tagging, and bandwidth burstability. Providers handle all the underlying transmission technology.
Costs for lit services include both setup charges and monthly recurring fees that scale with bandwidth. A 10 Gbps lit circuit costs roughly 10 times what a 1 Gbps circuit costs, whereas dark fiber costs stay relatively flat regardless of how much capacity you light.
Lit services make sense for moderate bandwidth requirements, shorter-term needs, or organizations that prefer operational simplicity over capital investment. Setup times are typically faster than dark fiber because you’re not installing and commissioning transmission equipment.
Service Comparison
| Factor | Dark Fiber | Lit Services |
| Upfront Cost | High (equipment purchase) | Low (provider equipment) |
| Monthly Cost | Fixed, bandwidth-independent | Scales with bandwidth |
| Flexibility | High (choose technology) | Moderate (provider-defined) |
| Technical Complexity | High (manage optical layer) | Low (provider managed) |
| Scalability | Excellent (upgrade endpoints) | Requires new circuits |
| Setup Time | Longer (install equipment) | Faster (plug and play) |
| Best For | High bandwidth, long-term | Moderate bandwidth, near-term |
Many organizations use a mix. Dark fiber between facilities with massive bandwidth requirements and long time horizons. Lit services for smaller sites or temporary connections where flexibility matters more than long-term cost optimization.
Latency Requirements for DCI Applications
Different applications have different latency requirements for multi-site connectivity. Understanding these requirements helps you select appropriate DCI solutions and facility locations.
Synchronous Data Replication
Synchronous replication writes data to storage at both sites before acknowledging completion to the application. This ensures zero data loss during failover – both sites always have identical data. But it requires low latency because every write operation waits for round-trip confirmation.
Most synchronous replication tolerates maximum 5-10 milliseconds round-trip latency. At higher latencies, the delays impact application performance noticeably. Storage vendors specify maximum distance or latency their synchronous replication supports – typically 50-100 miles or 5-10 milliseconds.
This latency requirement influences facility selection. If you need synchronous replication, your data centers must be relatively close together. Many organizations place primary and DR sites within the same metropolitan area or in adjacent markets to keep latency manageable.
Markets positioned strategically between other markets can enable synchronous replication across longer distances. A site in Kansas City maintains 12-15 millisecond latency to both coasts, potentially enabling synchronous replication across 1,000+ miles in each direction.
Asynchronous Data Replication
Asynchronous replication acknowledges writes locally before replicating to remote sites. This eliminates the application performance impact of replication latency but accepts potential data loss during failover – writes confirmed locally but not yet replicated would be lost.
Asynchronous replication tolerates much higher latency – 50-100 milliseconds works fine for most implementations. This allows placing sites further apart for better disaster recovery protection. Cross-country replication becomes practical where synchronous replication wouldn’t work.
The latency impacts recovery point objective (RPO) – how much data you can lose during failover. Lower latency means less data in flight at any moment, reducing potential loss. Higher latency increases data in flight and potential loss, though buffering and retry mechanisms help manage this.
Distributed Applications
Applications with components at multiple sites require DCI with latency appropriate to their chattiness. Applications making hundreds of requests between sites per user interaction need very low latency. Applications with looser coupling tolerate higher latency.
Database applications where application servers at one site query databases at another site need single-digit millisecond latency for good performance. Each query incurs round-trip latency, and this adds up quickly for chatty applications.
Object storage applications work well with higher latency because request patterns are typically less chatty. Retrieving large files tolerates higher latency better than making thousands of small requests.
Measure your actual application behavior to understand latency requirements. The theoretical architecture might suggest certain requirements, but actual testing reveals whether performance remains acceptable at various latency levels.
Financial Trading and Real-Time Systems
High-frequency trading requires microsecond-level latency, making only very short-distance DCI practical. Trading firms position infrastructure within miles of exchanges specifically to minimize latency.
Other financial applications have less stringent requirements but still need low latency. Risk management systems, order management systems, and market data distribution all benefit from DCI latency below 10 milliseconds.
Real-time communications and collaboration tools tolerate 100-150 milliseconds before quality degrades noticeably. This makes cross-country DCI practical for these applications even though it wouldn’t work for trading systems.
DCI for Disaster Recovery and Business Continuity
Disaster recovery is one of the most common drivers for DCI deployment. Organizations need geographic diversity to protect against regional outages while maintaining the connectivity required to keep data synchronized and enable failover.
Geographic Diversity Requirements
Effective disaster recovery requires distance between sites. Placing primary and DR facilities too close together leaves both vulnerable to the same regional disasters – hurricanes, earthquakes, and utility grid failures affecting entire metros.
The industry rule of thumb suggests 100+ miles between facilities for true disaster recovery. This distance provides protection against most regional events while keeping latency manageable for replication. Some organizations stretch this to 200-500 miles or more for better protection.
Strategic positioning balances disaster recovery protection with latency requirements. Sites 500 miles apart provide excellent geographic diversity while maintaining latency under 10 milliseconds for synchronous replication. Sites 1,000 miles apart require asynchronous replication but provide better disaster protection.
Hot, Warm, and Cold DR Sites
Hot DR sites run identical configurations to production, with data continuously replicated and systems ready to assume load instantly. This requires substantial DCI bandwidth to keep sites synchronized and minimal latency for replication.
Warm sites maintain infrastructure ready to run, but don’t actively serve production load. Data replicates to warm sites, but systems aren’t processing production transactions. Failover requires bringing systems online, but happens relatively quickly because data is current and infrastructure exists.
Cold sites provide space and basic infrastructure, but don’t maintain current data or running systems. Recovery requires restoring data from backups and configuring systems from scratch. Cold sites need minimal DCI – just enough to transfer backup data periodically.
The DR strategy determines DCI requirements. Hot sites need high-bandwidth, low-latency circuits for continuous replication and potential active-active operation. Cold sites can use much cheaper connectivity because they’re not maintaining real-time synchronization.
Failover and Failback Procedures
DCI enables automated failover that redirects traffic to DR sites when primary sites fail. This requires monitoring systems that detect failures, failover mechanisms that redirect traffic, and network connectivity to make the DR site accessible to users.
Planned failovers for maintenance or testing need to happen without disrupting users. This is where DCI really proves its value – the connectivity between sites allows graceful migration of workloads rather than just hard failover during disasters.
Failback after recovering from outages can strain DCI circuits if large amounts of data are changed at the DR site during failover. Plan for reverse replication capacity to rebuild the primary site without impacting production performance at the DR location.
Testing Without Impact
The best DR plan fails if you’ve never tested it. Regular DR tests verify procedures work, train staff, and identify gaps in documentation. But testing often impacts production operations if it requires taking over the DR site or redirecting traffic.
DCI with adequate capacity enables testing that doesn’t impact production. Replicate production workloads to the DR site, fail them over, verify they work correctly, then fail back – all while production continues running at the primary site. This validation builds confidence without the risk of full production cutover.
Test both failover and failback procedures. Organizations often test failover but neglect failback, then discover failback is more complex because data at the DR site diverged from the primary during the test.
Strategic Data Center Interconnect Routes and Mid-Country Hubs
Where you locate facilities and how you route DCI circuits impacts both performance and cost. Strategic planning of facility locations and circuit routing optimizes this architecture.
Mid-Country Aggregation
Facilities in mid-country locations serve as efficient aggregation points for DCI networks. A hub in Kansas City connects to both coasts with roughly equal latency and potentially lower combined circuit costs than direct coast-to-coast connections.
This hub-and-spoke approach reduces circuit count compared to full mesh. In a four-site deployment, hub-and-spoke requires three circuits while full mesh needs six. The cost savings can be substantial, though traffic between spoke sites incurs an extra hop through the hub.
The latency penalty for hub-and-spoke depends on the hub location. A well-positioned hub adds minimal latency compared to direct routing. Traffic from Northern Virginia to Northern California through a Kansas City hub might see 25-30 milliseconds total versus 30-35 milliseconds for a direct circuit – not enough difference to matter for most applications.
Avoiding Common Routing Mistakes
Don’t assume geographic straight lines represent actual fiber paths. Fiber follows rights-of-way, which means routes might detour substantially from direct paths. Check actual fiber routing before assuming latency based on straight-line distance.
Watch for circuits that piece together multiple segments. Sometimes providers build “circuits” by stitching together capacity from multiple underlying systems. These can introduce unexpected latency or reliability issues compared to clean single-segment routes.
Verify your circuits don’t share common physical paths for diversity. Two circuits that route through the same conduit don’t provide redundancy, regardless of whether they use different providers. Physical diversity requires different rights-of-way.
Multi-Cloud and Hybrid Connectivity
Organizations running hybrid cloud architectures need DCI that works with cloud connectivity. Place colocation facilities where they can establish both DCI to your other sites and direct connectivity to AWS, Azure, or Google Cloud.
Not every market offers good cloud connectivity options. Verify facilities in target markets support the cloud on-ramps you need before committing to locations. The best carrier-neutral facilities offer both strong DCI routing options and multiple cloud connectivity choices.
Planning for Growth
Start with circuits sized for current needs plus reasonable growth headroom. Upgrading circuits later is possible, but often requires installation fees and potentially new equipment. Starting with 10 Gbps when you need 3 Gbps costs more monthly, but avoids upgrade hassles when traffic grows.
For dark fiber deployments, consider future capacity needs. Installing extra fiber pairs during initial construction costs relatively little but gives you expansion options later. Adding fiber afterward often means expensive and disruptive construction work.
Ready to Connect Your Infrastructure the Right Way?
Data center interconnect is what makes multi-site infrastructure actually work. Without proper DCI, you’re running isolated islands of infrastructure that can’t effectively support disaster recovery, distribute workloads, or provide the geographic diversity serious applications need.
The organizations that get DCI right think strategically about facility location, circuit routing, and technology selection rather than just connecting whatever sites they happen to have. They place facilities where strong fiber infrastructure exists, they route circuits over proven paths with good latency profiles, and they size capacity for both current needs and reasonable growth.
Getting this wrong means either overpaying for connectivity you don’t fully need or struggling with insufficient capacity, high latency, or reliability issues that impact applications. The key is matching DCI solutions to actual requirements rather than defaulting to whatever options seem easy.
Ready to implement DCI that delivers the performance and reliability your applications actually need? Netrality Data Centers operates facilities in strategic markets, including Kansas City, Philadelphia, and Houston, with access to diverse fiber routes and carrier-neutral interconnection that enables cost-effective DCI deployment. Our positioning in mid-country and regional markets provides optimal routing for multi-site architectures while maintaining the network resiliency enterprises require. Contact our team to discuss your multi-site connectivity requirements and explore how strategic facility selection and routing can optimize your DCI costs and performance.
