Data Center Network Design for Resilience and Maintainability

Start from requirements

• Number of servers and racks today and in three to five years.
• Traffic patterns: mostly users to servers, or heavy traffic between servers and storage.
• Storage networking needs, such as iSCSI or NVMe over TCP.
• Availability requirements for the services hosted.
• The skills of the team that will run it.

Choosing an architecture

A leaf-spine fabric normally routes between switches using equal-cost paths, as described in RFC 7938, and often adds an EVPN-VXLAN overlay to provide flexible Layer 2 and Layer 3 segments without large spanning-tree domains.

Redundancy

• Connect servers to two switches using link aggregation to a switch pair, or active/standby network interfaces.
• Use at least two spines or two core switches.
• Prefer routed links with fast failure detection over stretched Layer 2 designs.
• Run firewalls in high-availability pairs connected to different switches.
• Feed every rack from two independent power paths, and dual-power every device.

Design for maintenance

Hardware fails and software needs upgrading. A maintainable design lets engineers drain traffic from one switch, upgrade it, check it and return it to service while the other carries the load. Standard configuration templates, agreed software versions and pre- and post-change checks make that routine instead of risky.

Segmentation

• Separate production, development, management, storage and internet-facing zones, using VRFs where appropriate.
• Filter traffic between zones with firewall policy.
• Use microsegmentation in virtualization platforms for finer control between workloads.

Storage traffic

Keep storage traffic on dedicated VLANs or interfaces. Enable lossless Ethernet features only when the storage vendor requires them, and apply jumbo frames consistently end to end or not at all.

Out-of-band management

• Provide a separate management network for switch management ports and server consoles.
• Add console servers for serial access, with an independent connection such as cellular.
• Protect out-of-band access with MFA and a jump host, and log every session.

Automation and configuration management

• Keep a source of truth for devices, IP addresses, VLANs and cabling.
• Back up configurations automatically and compare them for unexpected changes.
• Generate configurations from templates and review changes before applying them.

Physical layer

Plan rack layouts, power capacity, cooling with hot and cold aisles, and structured cabling with consistent labeling. Neat, documented cabling reduces mistakes during changes and speeds fault finding.

Documentation to keep current

• Physical diagrams: racks, device positions, power feeds and cable paths.
• Logical diagrams: VLANs, VRFs, routing, firewall zones and overlays.
• An IP address plan and a record of which switch ports connect to which servers.
• Standard configuration templates and approved software versions.
• Procedures for common changes, upgrades and recovery from failures.
• Support contracts, serial numbers and escalation contacts.

Common mistakes

• Dual-homed servers whose two cables both run to the same switch.
• Stretching Layer 2 networks between rooms or sites without a clear need.
• Carrying management traffic on production networks.
• Software versions drifting apart between the two switches of a pair.
• Redundant power supplies plugged into the same power strip.
• A design that only its original engineer knows how to operate.

Monitor and test

• Monitor interface errors, buffer drops, routing adjacencies, temperature and power.
• Before production, fail each switch, link and power supply in turn and record the result.
• Repeat failover tests after major upgrades.