White Paper: Using CFD for Data Centers
Extract from the EOLIOS white paper on Data Centers
Continue browsing :
Our latest news:
Our projects :
Our fields of intervention :
The thermal load in data centers has significantly increased in recent years. And this trend continues, because there is a process of reduction of the size of the electronic equipment with a simultaneous increase in computing power , which leads to the release of a large amount of heat per unit (one unit of server rack height). If a few years ago it seemed that the cooling capacity of 5 kW per server rack was quite sufficient to cover all the existing and expected needs of customers in the near future, today there are equipment on the market that even if not the entire server rack is filled, emits more than 10 kW.
Almost all server manufacturers have such equipment. The large groups offer A unified IT system, which achieves a 6U power consumption of 2 kW or more depending on the operating mode. It is not uncommon that more than 10 kW of heat can be generated per rack, we observe racks up to a power of 45 kW.
Server virtualization results in increased heat dissipation from a single processor
The use of server virtualization technology “makes it worse”. Virtualization of servers can significantly increase their load, and if the old processors were idle for 75-85% of the time, then via the use of virtualization, the processor load in the servers increases significantly and, therefore, more heat is generated on a server. ASHRAE’s Best Practices for Energy Efficiency in Data Center Facilities confirms this data.
The heat output produced by the rack is constantly increasing.
Therefore, when designing a modern data center, it is necessary to focus on thermal loads of 10 kW or more per server rack. Or, as a last resort, it is necessary to allocate specific zones in the data center’s computer room, which will provide cooling from 10 kW per server rack.
To get a sufficiently accurate answer to this question, the designer’s use of only the heat balance equation with the addition of a 10% to 20% cooling capacity margin and an Excel program is no longer sufficient.
A number of problems in data center operations
A number of problems also arise when operating an existing data center, even with low thermal loads. For example, after installing additional equipment, dead zones may appear in the data center.
Local areas of equipment overheating (called hot spots) or, on the contrary, areas of low temperature can occur in data centers (for natural cooling systems).
Clearly, overheating servers, data storage systems, network and telecommunications equipment is bad; sooner or later, high temperatures will cause failures , and therefore potential data loss.
On the other hand, hygrometry also negatively affects the functioning of servers and data storage systems. Low temperatures lead to increased humidity, which can result in condensation. The relative humidity, according to ASHRAE, should not exceed 80%. Some manufacturers build temperature and relative humidity sensors into servers and storage systems, and software controls can shut down the hardware when humidity and temperature limits are exceeded.
The temperature in server rooms, where IT equipment is installed and operated, is limited by standards not only by the upper limit, but also by the lower limit. According to the latest ASHRAE TC 9.9 technical committee requirements published in 2016, the temperature in the server room should not be lower than 18 degrees Celsius. In addition, the lower temperature leads to inefficient use of electricity, resulting in higher data center operating costs .
To combat local hot spots, the customer may be required to install floor fans near the server racks, mount additional air or liquid cooling units (if, of course, there is room for their installation). However, the use of such a “radical” method is not necessarily a necessity. But sometimes it turns out that only one thing should have been replaced: it may be a matter of removing or adding grates from the raised floor and the problem can be solved. However, it is extremely difficult to identify bottlenecks without special software tools and precise knowledge of thermo-aerodynamic effects. Often, the customer uses data center space extremely inefficiently, without fully loading the cabinets and distributing the equipment evenly (if it can be done at all). But all this could have been avoided by creating a CFD thermodynamic model of the data center and performing optimization calculations according to this model.
What is the height of the raised floor in a data center?
When designing a new data center, an architect in charge of the project always has a question: how high should a raised floor be in a data center?
It is clear that the more The higher the height of the false floor, the better the airflow resistance, plus various networks (water networks, cable trays and cables) can be stored, as well as additional structures and equipment can be placed under the raised access floor, for example, power distribution networks or consolidation points of a structured cabling system…
However, as the height of the raised floor increases, the cost of the building structure will increase and, by the way, the space between the raised floor and the ceiling will decrease, which can complicate the creation of a duct system to supply warm air to the cooling units (CRAH). A few years ago, a recommendation was published on the height of the raised floor in relation to the size of the data center’s machine room.
With engine room area up to 70 m², the height of the raised floor must be at least 400 – 500 mm, if the surface of the room is more than 100 sqmthe height of the raised floor must be at least 500 – 700 mm, if the engine room is more than 300 m², the height of the raised floor must beat least 700mm. This rule of thumb worked when the load per rack did not did not exceed 5 kW and that the hot and cold air insulation technology was not used (separation between hot and cold aisle). In order to get a precise answer to the question of the height of the raised floor, it is recommended to perform a CFD simulation of the air flows, to calculate several options and to choose the most appropriate.
What is CFD simulation?
CFD is an acronym for Computational Fluid Dynamics
With the help of specialized software, the user creates a three-dimensional model of an object, imposes certain boundary conditions, selects models representing physical phenomena occurring in gaseous and liquid media (heat transfer, media flow, thermal conductivity, radiation, convection, etc.), selects a calculation method and performs calculations ), selects a calculation method and performs calculations … On the basis of the obtained calculation results, the user evaluatesif necessary, modifies the computer modeland performs the calculations again. The purpose of modeling is to write the physical phenomena as well as possible and then to find an adapted and satisfactory solution to the design problems that may be encountered.
The results of the simulation are used in the design decision making process to further improve the created model of the plant, to identify bottlenecks in the operating plant and to optimize the operating system.
Using CFD simulation in data centers
Unfortunately, higher operating temperatures can lead to reduce the reaction time in case of a rapid increase in temperature due to a failure of the cooling unit. A data center containing servers operating at higher temperatures may experience instantaneous hardware failures simultaneous. Recent ASHRAE regulations emphasize the importance of proactive monitoring of environmental temperatures inside server rooms.Data centers are ideal objects for computer modeling because it is impossible to create a prototype or physical model of a data center. And without creating a data center model, it is impossible to predict with sufficient accuracy how the air conditioning system will perform in a real operating facility, how the air conditioning system will behave when the load changes, how the temperature will change in a row of server racks and along the height of each rack.
When designing an air conditioning system in a data center, a large number of parameters must be taken into account. We will give here some of them:
- The size and the volume of the room;
- Placement of telecommunications cabinets and racks in the server room;
- Floor height raised; direction, volume and speed of cold air current movement;
- Location of air conditioning equipment;
- The types of fans used and the direction of the air flow,
- Consideration of obstacles to the flow of cold air;
- The type of raised floor slabs used and the geometry of the outlets.
When designing an air conditioning system without the use of CFD analysis, most of these parameters are not properly taken into account or are overdimensioned. Indeed, the real influence on the distribution of temperature and humidity in the data center room according to the studied parameter cannot be reliably estimated without accurate computer simulation.
CFD software for data centers
There are a large number of programs on the market that solve various problems related to the simulation of flows of liquids and gases. Among these programs are the following: ANSYS, Autodesk CFD, Xflow, Open Foam, Phoenics, Flow Vent,STAR-CD , FASTEST-3, Flow Vision, Tile Flow, Sigma6room, Gas Dynamics Tool… However, not all thermodynamic flow simulation programs have ready-to-use modules and built-in element libraries that take into account the specificities of data centers.
Software such as Tile Flow and Sigma 6 have built-in modules, programs and libraries to simulate airflow in the data center. For engineers not used to working with CFD modeling programs, it makes sense to consider purchasing this type of software, which already contains ready-to-use models for calculating air flows in the data center, there are libraries of equipment (e.g. fans, pumps, air conditioning units). In all cases, what makes the quality of the study is the experience level of the engineer in charge of the simulation. CFD engineering must be done by specialists.
Steps in the design of CFD modeling of a data center
Recovery of study assumptions
Before performing the modeling process for an existing data center, it is necessary to conduct a complete and precise study of the object: measure air flow velocity, measure pressure, take temperature measurements, determine air flow channels and detect obstacles and possible locations of air leaks. In other words, the task of examining an existing object itself is quite laborious but, nevertheless, extremely useful. Because in the data collection process, bottlenecks are identified. To solve the problem of creating a model for a new data center, it is necessary to collect initial data from the space and validate assumptions about the technologies and devices used.
3D modeling of the data center
Next, create a geometric model of the data center (or digital twin) and the elements that make up the data center. A 3D model of an object is created using CAD programs, then the data is exported to the CFD simulation module.
Creation of the mesh
Next, the resolution model is created. This step is performed in programs using integrated mesh generation software modules or by using separate software products. The accuracy, the convergence and the speed of calculation depend on the mesh . The quality of the results obtained depends directly on the quality of the mesh (fineness, mesh adaptation…). After the mesh construction step, the user has to check the quality of the constructed mesh by different parameters (asymmetry of the elements, ratio height / width).
Implementation of boundary conditions & simulations
Boundary conditions are entered into the program and models are selected according to the assumptions, then a calculation is performed, which may converge or diverge (i.e. not have a correct solution) according to the different parameters above.
After convergence, the results of the calculations can be processed by special programs and displayed in graphical, tabular or evenanimated form, clearly demonstrating the changes in the physical parameters. For data centers, a visual representation of calculated data is usually used in the form of temperature distribution over the area of the computer room and the height of the server racks.
Then the engineer analyzes the calculated results and, if necessary, modifies the object models and performs the calculations again.
What impact on the design?
Modern design tools allow CFD engineers to interact with the various trades, in order to simply explain the phenomena at the origin of the problems and then to propose solutions that can be validated by all parties.
CFD programs can simulate the flow of liquids and gases, as well as other physical phenomena associated with this process, such as heat transfer. Thermodynamic modeling offers great opportunities to analyze the flow of liquids and gases, which allows to design at a high professional level new systems, equipment or tooptimize the operation of existing systems.
Without the use of CFD modeling, it is impossible to get accurate answers to such fundamental questions as the distribution of temperature and humidity along the cold aisles and the height of the server rack room, depending on :
- heat load;
- of the installation sites of the air conditioning units;
- temperature of coolants and coolants;
- raised floor height ;
- the distribution of broadcasting schedules;
- the type of fans and other parameters.