Airflow Options for Cisco MDS 9396S SAN Switch This document describes how to configure, administer and troubleshoot airflow options in the MDS 9396S switch. Anyone can use this document along with hardware installation guide of MDS 9396S under power supply section. 2016 Cisco and/or its affiliates. All rights reserved. 1
Introduction In a Data Center, airflow is a very important aspect to maintain the steady temperature across hot and cold aisles. Director class switches have their ports on the front side and hence in most cases, switches will connect its cables to patch panel or top-of-rack switch from the front side. In this case, the preferred airflow will be front-to-back to match the airflow direction of larger switches. On a rack with servers, the network ports for servers are in the back and hence the switch direction has to be reversed to allow smooth and direct cabling from servers. Here, the cabinet will use front-to-back airflow as well but note that switch is in reverse direction. If the switch cannot support reversible airflow, it can conflict with existing airflow and can create temperature issues across hot and cold aisles. When Cisco MDS 9396S multilayer fabric switch was first released in mid-2015, only port-side exhaust (back-tofront) airflow was supported using unidirectional port-side exhaust airflow support for fan trays and power supply units (PSU). With the introduction of NX-OS 6.2(15) software release and new a fan tray along with a new bidirectional airflow power supply unit, it now offers a choice of airflow front-to-back (Port-side intake) or back-to-front (Port-side exhaust). In this document, Front side refers to the Port side and Back/Rear side refers to the side with the Power Supply and fan tray as shown in the following pictures. Picture 1. Front View of MDS 9396S (Port-side) System LEDs Cisco MDS 9396S Multilayer Fabric Switch Console Front View USB Port Group Management Port 48 x 16G FC Line-rate Ports Picture 2. Rear View of MDS 9396S (Back/Rear-side) Color Codes PSU 1 FAN Tray 1 FAN Tray 2 PSU 2 Hot Swappable Fan Trays Hot Swappable Power Supply Units (PSU) Reversible Airflow Selection Cisco MDS NX-OS Release 6.2(15) now supports new bidirectional airflow PSUs and port-side intake fan trays and reversible (port-side exhaust airflow and port-side intake) airflow options. To achieve this, we need the right set of two key components: Power Supply Unit (PSU) and Fan Tray. Cold Air Hot Air 2 RU Hot Air Port Side Exhaust Airflow (Back-to-Front Airflow, Originally Shipped) Cold Air Port Side Intake Airflow (Front-to-Back Airflow, New Option) 2016 Cisco and/or its affiliates. All rights reserved. 2
Power Supply Unit (PSU) Cisco MDS 9396S now supports two different power supply units: Part Number Description Color Comments DS-CAC- 1200W-E DS-CAC-1200W Cisco 1200W AC Power Supply (port-side exhaust) Cisco 1200W AC Power Supply (bidirectional) (HVAC certified) Blue White Port-side exhaust airflow only Bi-directional airflow (portside intake and port-side exhaust airflow) Each PSU has a power receptacle and a status LED on the faceplate, and a handle for inserting and removing the unit from the chassis. PSUs are denoted with white or blue color on the side handle of PSU to identify the airflow direction supportted. Blue color PSU: This PSU supports port-side exhaust airflow direction only. Port-side intake Fan trays are not allowed with this PSU. If system detects port-side intake fan tray with this PSU installed, it will trigger an alarm and will automatically shut down in specified interval. White color PSU: This PSU supports bidirectional airflow. Depending on the airflow of the system fan, it can work in port-side exhaust airflow mode or port-side intake airflow mode. The system will automatically match the airflow with the other PSU and system fan to avoid any conflicts. This PSU is also HVAC (High Voltage AC)/HVDC (High Voltage DC) tested and certified by Cisco. PSUs are hot swappable and can be replaced individually (within certain timeframe) while in operations, without affecting the system performance. In the event of a PSU or AC supply (in grid redundant mode) failure, the system will continue to run. Procedures for installing and removing PSUs are detailed in the Removing and Installing AC Power Supplies section. Fan Tray Modules Cisco MDS 9396S supports two fan trays, one with port-side intake airflow and other one with port-side exhaust airflow support. To match the airflow of PSUs, the fan trays have to match the direction of airflow as well. The fan trays have colored stickers on them to identify the airflow support. Part Number Description Color Comments DS-C96S- FAN-E= DS-C96S- FAN-I= Cisco MDS 9396S Fan Tray (port-side exhaust) Cisco MDS 9396S Fan Tray (port-side intake) Blue Red Port-side exhaust airflow only Port-side intake airflow only Note that one of the Power Supply Unit (PSU) supports bidirectional airflow (port-side exhaust or port-side intake) but for Fan trays, there is either a port-side exhaust fan tray or a port-side intake fan tray. There are various combinations of fan trays and power supply modules to support port-side intake/port-side exhaust airflow in the same chassis. The minimum software requirement to support reversible airflow is NX-OS Release 6.2.(15). This is not supported on Cisco NX-OS Release 7.3(0)D1(1) but subsequent software releases planned to be released in 2016 will support this configuration. 2016 Cisco and/or its affiliates. All rights reserved. 3
Table below describes possible combinations of fan trays, PSUs and their airflow directions along with their support matrix. Option Fan #1 Fan #2 PSU #1 PSU #2 Minimum NX-OS Required System Response 1 Port-side exhaust Port-side exhaust Port-side exhaust PSU Port-side exhaust PSU NX-OS 6.2(13) Valid Configuration 2 Port-side intake Port-side intake Bi-directional PSU Bi-directional PSU NX-OS 6.2(15) Valid Configuration 3 Port-side exhaust Port-side exhaust Bi-directional PSU Bi-directional PSU NX-OS 6.2(15) Valid Configuration 4 Port-side exhaust Port-side exhaust Port-side exhaust PSU Bi-directional PSU NX-OS 6.2(15) Valid Configuration 5 Port-side exhaust Port-side exhaust Bi-directional PSU Port-side exhaust PSU NX-OS 6.2(15) Valid Configuration 6 Port-side exhaust Port-side intake Any PSU Any PSU NX-OS 6.2(15) Invalid configuration* 7 Port-side intake Port-side exhaust Any PSU Any PSU NX-OS 6.2(15) Invalid configuration* 8 Port-side intake Port-side intake Port-side exhaust PSU Any PSU NX-OS 6.2(15) Invalid configuration* 9 Port-side intake Port-side intake Any PSU Port-side exhaust PSU NX-OS 6.2(15) Invalid configuration* * If system is powered up, it will not allow the board to Power ON state and STATUS LED starts blinking red. If the switch is already ON, STATUS LED starts blinking red, System will enter shutdown mode in 10 minutes maximum time. Parts Replacement FAN tray replacement: To replace the FAN tray, it should match the existing airflow direction. The port-side intake/port-side exhaust fan tray can only be swapped with matching airflow directions. Power Supply replacement: To replace the damaged PSU, use portside exhaust or bidirectional PSU. If the airflow is port-side exhaust, replace using port-side exhaust PSU or bidirectional PSU. For port-side intake airflow direction, it can only be replaced using bidirectional airflow PSU. The airflow direction using bi-directional PSU will be detected by software and will modify the PSU airflow accordingly. The software will detect the direction of airflow in the FAN tray and will modify the airflow in Power supply to make it port-side intake. To avoid any unexpected damage to hardware, the correct combination of power supplies and Fan trays should be installed with matching airflows. During power-on process, MDS 9396S system board will perform Power-On Self-Test (POST) function and will check the airflow directions of the fan trays and PSUs. The system board will correct the airflow of bidirectional PSUs to match the system airflow but if it finds any other mismatch in airflow directions, the system board will not power on and will set STATUS LED to blinking red. Any incompatibility issues found during or after POST will be reported as an alarm and chassis might perform automatic shutdown to prevent further damage to the unit. If system finds any of the invalid configurations mentioned above, it will automatically start system shutdown process in 10 minutes, regardless of temperature within the chassis. If the temperature within chassis crosses the Major Threshold value before the 10 minutes interval, it will start shutdown procedure immediately regardless of the timer. 2016 Cisco and/or its affiliates. All rights reserved. 4
The default airflow of MDS 9396S is port-side exhaust. The software requirement to support port-side intake mode is NX-OS 6.2(15) or later releases of 6.2.x train. If the system is running in port-side intake mode with respective fan trays along with PSU configurations, it will expect the replacement PSU as well to match the airflow direction. In this case, user can only use bidirectional PSU. Whenever a new bi-directional PSU is used as a replacement in the chassis, it will first start with port-side exhaust airflow direction, one might see an alarm in the logs saying Airflow conflict is detected. This conflict will be corrected in less than 15 seconds and PSU will change the airflow to port-side intake mode automatically. Here are the logs observed during this test procedure after inserting bidirectional airflow PSU in PS1 slot with both fan trays and PSU2 running in port-side intake mode: MDS9396S-174130# sh env fan Fan: Fan Model Hw Status Airflow direction ChassisFan1 FAN Module 1 -- Ok Front to Back ChassisFan2 FAN Module 2 -- Ok Front to Back Fan_in_PS1 -- -- Ok Back to Front Fan_in_PS2 -- -- Ok Front to Back CLI Logs: MDS9396S-174130# MDS9396S-174130# 2016 Jun 21 12:23:57 MDS9396S-174130 Jun 21 12:23:57 %KERN-0-SYSTEM_MSG: [496944.553395] Airflow conflict is detected. Incompatible PSU/ Fan found. Switch will be shutdown if compatible PSU/ Fan is not installed. - kernel 2016 Jun 21 12:23:57 MDS9396S-174130 %PLATFORM-5-PS_FOUND: Power supply 1 found (Serial number ****) 2016 Jun 21 12:23:57 MDS9396S-174130 %PLATFORM-2-PS_OK: Power supply 1 ok (Serial number ****) 2016 Jun 21 12:23:57 MDS9396S-174130 %PLATFORM-5-PS_STATUS: PowerSupply 1 current-status is PS_OK 2016 Jun 21 12:23:57 MDS9396S-174130 %PLATFORM-2-PS_FANOK: Fan in Power supply 1 ok 2016 Jun 21 12:24:01 MDS9396S-174130 Jun 21 12:24:00 %KERN-0-SYSTEM_MSG: [496947.445399] Airflow conflict is corrected. - kernel MDS9396S-174130# MDS9396S-174130# sh env fan 2016 Cisco and/or its affiliates. All rights reserved. 5
Fan: Fan Model Hw Status Airflow direction ChassisFan1 FAN Module 1 -- Ok Front to Back ChassisFan2 FAN Module 2 -- Ok Front to Back Fan_in_PS1 -- -- Ok Front to Back Fan_in_PS2 -- -- Ok Front to Back Reference table for fan tray module or PSU replacement in MDS 9396S: Airflow Fan #1 Fan #2 Minimum NX-OS Release Replacement FAN Tray Module Port-side exhaust Port-side exhaust Port-side exhaust NX-OS 6.2(13) Port-side exhaust FAN tray Port-side exhaust Port-side exhaust Port-side exhaust NX-OS 6.2(15) Port-side exhaust FAN tray Port-side intake Port-side intake Port-side intake NX-OS 6.2(15) Port-side intake FAN tray Airflow PSU #1 PSU #2 Minimum NX-OS Release Replacement PSU Port-side exhaust Port-side exhaust PSU Port-side exhaust PSU NX-OS 6.2(13) Port-side exhaust PSU Port-side exhaust Port-side exhaust PSU Port-side exhaust PSU NX-OS 6.2(15) Port-side exhaust or Bi-directional PSU Port-side exhaust Bi-directional PSU Port-side exhaust PSU NX-OS 6.2(15) Port-side exhaust or Bi-directional PSU Port-side intake Bi-directional PSU Port-side intake PSU NX-OS 6.2(15) Bi-directional PSU Port-side intake/exhaust Bi-directional PSU Bi-directional PSU NX-OS 6.2(15) Bi-directional PSU 2016 Cisco and/or its affiliates. All rights reserved. 6
Here are some of the common questions, use cases and issues to help with selecting right set of hardware and software. Frequently Asked Questions 1. My MDS 9396S is currently running NX-OS 6.2(15) with port-side intake configuration and/or bidirectional PSU. Can I downgrade to NX-OS 6.2(13) now? Answer: Bidirectional PSU and Port-side intake FAN tray are not supported under NX-OS 6.2(13). If at any point, MDS 9396S is required to downgrade to NX-OS 6.2(13) release, please replace the PSU and Fan tray with port-side exhaust airflow directions and follow the downgrade process defined here. 2. My MDS 9396S is running NX-OS 6.2(15) now. Can I use the new fan trays and/or PSU to reverse the airflow? Answer: Yes, if MDS 9396S is running NX-OS 6.2(15), it will now detect the bidirectional PSU and port-side intake fan tray correctly. Hence you can now swap the fan tray and PSU to your choice. 3. How much time do I have to replace PSU/fan tray in the switch? Answer: Cisco MDS 9396S has 23 temperature sensors across the switch. It is designed to shutdown automatically to avoid any damage to ASICs and other internal boards due to high temperatures. Depending on the temperature threshold, it may take few seconds to few minutes to initiate shutdown process. You can monitor the switch temperature using CLI command show environment temperature. 4. How much time will the switch keep running with incorrect airflow (combination of port-side intake and port-side exhaust airflow)? Answer: 10 minutes OR temperature threshold alarm whichever comes first. System is designed to initiate shutdown process in 10 minutes even, if it does not hit the temperature alarm. If it hits the temperature threshold alarm before that, it will shutdown immediately. 5. Can I use port-side intake fan trays with port side exhaust PSUs in this switch? Answer: No. This is invalid configuration. Switch will initiate shutdown process in 10 minutes if the situation is not corrected by replacing it with the correct parts. Table above shows some of the valid configurations. The switch is designed to initiate shutdown process in 10 minutes if it finds conflicting airflow in PSU or Fan tray. 6. Can I mix and match the port-side exhaust and bidirectional PSU in MDS 9396S? i.e. PSU1 is port-side exhaust and PSU 2 is bidirectional Answer: Yes, using NX-OS 6.2(15), we can mix and match the PSUs the way we want it. We can use port-side exhaust and bidirectional PSUs in any slot. The airflow, though will be port-side exhaust only. 7. Why do I see Airflow conflicting error message after replacing failed PSU with bidirectional airflow PSU? Answer: By default, MDS 9396S will power ON the PSU in portside exhaust mode. When it finds that system is running with portside intake airflow, it will correct the airflow in first 15 seconds of powering ON and will clear the alarm as well. Additional Resources Cisco MDS 9396S 16G Multilayer Fabric Switch Data Sheet Cisco MDS 16G Multilayer Fabric Switch AAG MDS 9396S Hardware Installation Guide 2016 Cisco and/or its affiliates. All rights reserved. Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this URL: www.cisco.com/go/trademarks. Third-party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (1110R) C17-737490-00 07/16