White Paper: Using CFD for Data Centers
Extract from the EOLIOS white paper on Data Centers
- A passionate team
- Exclusive estates
- All sectors
- Data Center
- A passionate team
Continue navigation :
Our latest news :
Our projects :
Our areas of expertise :
Technical files :
White paper
The thermal load in data centers has significantly increased in recent years. And this trend continues, because there is a process of reduction in the size of electronic equipment with simultaneous increase in computing power, leading to the release of a large amount of heat per unit (one unit of server rack height). If a few years ago it seemed that the cooling capacity of 5 kW per server rack was quite sufficient to cover all the existing and expected needs of customers in the near future, today there are equipment on the market that even if not the entire server rack is filled, generates more than 10 kW.
Almost all server manufacturers have such equipment. The large groups offer A unified IT system, which achieves a 6U power consumption of 2 kW or more depending on the operating mode. It is not uncommon that more than 10 kW of heat can be generated per rack, we observe racks up to 45 kW.
Server virtualization leads to increased heat dissipation from a single processor
The use of server virtualization technology “makes it worse”. Virtualization of servers can significantly increase their load, and if the old processors were idle for 75-85% of the time, then via the use of virtualization, the processor load in the servers increases significantly and, therefore, more heat is generated on a server. ASHRAE’s Best Practices for Energy Efficiency in Data Center Facilities confirms this data.
Heat production per rack is constantly increasing.
Therefore, when designing a modern data center, it is necessary to focus on thermal loads of 10 kW or more per server rack. Or, as a last resort, it is necessary to allocate specific zones in the data center’s computer room, which will provide cooling from 10 kW per server rack.
To obtain a sufficiently precise answer to this question, the designer’s use of a heat balance equation alone, with the addition of a cooling capacity margin of 10% to 20% and an Excel program, is no longer sufficient.
A number of problems in data center operations
A number of problems also arise when operating an existing data center, even with low thermal loads. For example, after installing additional equipment, dead zones may appear in the data center.
Local areas of equipment overheating (known as hot spots) or, on the contrary, low-temperature zones can occur in data centers (for natural cooling systems).
It’s clear that overheating servers, data storage systems, network and telecommunications equipment is bad; sooner or later, high temperatures will lead to breakdowns, and therefore potential data loss.
On the other hand, hygrometry also negatively affects the functioning of servers and data storage systems. Low temperatures lead to increased humidity, which can result in condensation. The relative humidity, according to ASHRAE, should not exceed 80%. Some manufacturers build temperature and relative humidity sensors into servers and storage systems, and software controls can shut down the hardware when humidity and temperature limits are exceeded.
The temperature in server rooms, where IT equipment is installed and operated, is limited by standards not only by the upper limit, but also by the lower limit. According to the latest ASHRAE TC 9.9 technical committee requirements published in 2016, the temperature in the server room should not be lower than 18 degrees Celsius. In addition, the lower temperature leads to inefficient use of electricity, resulting in higher data center operating costs .
To combat local hot spots, the customer may be required to install floor fans near the server racks, mount additional air or liquid cooling units (if, of course, there is room for their installation). However, the use of such a “radical” method is not necessarily a necessity. But sometimes it turns out that only one thing should have been replaced: it may be a matter of removing or adding grates from the raised floor and the problem can be solved. However, it is extremely difficult to identify bottlenecks without special software tools and precise knowledge of thermo-aerodynamic effects. Often, the customer uses data center space extremely inefficiently, without fully loading the cabinets and distributing the equipment evenly (if it can be done at all). But all this could have been avoided by creating a CFD thermodynamic model of the data center and performing optimization calculations according to this model.
How high is the raised floor in a data center?
When designing a new data center, the architect in charge of the project always has a question: how high should a raised floor be in a data center?
Clearly, the greater the height of the raised access floor, the greater the resistance to air flow, the more diverse the networks (water networks, cable trays and cables) that can be stored, and the more additional structures and equipment that can be placed beneath the raised access floor, such as power distribution networks or consolidation points for a structured cabling system…
However, as the height of the raised floor increases, the cost of the building structure will increase and, by the way, the space between the raised floor and the ceiling will decrease, which can complicate the creation of a duct system to supply warm air to the cooling units (CRAH). A few years ago, a recommendation was published on the height of the raised floor in relation to the size of the data center’s machine room.
With engine room area up to 70 m², the height of the raised floor must be at least 400 – 500 mm, if the surface of the room is greater than 100 m², the height of the raised floor must be at least 500 – 700 mm; if the machine room is over 300 m², the height of the raised floor must beat least 700 mm. This rule of thumb worked when the load per rack did not exceed 5 kW, and when hot and cold air insulation technology was not used (separation between hot and cold aisles). In order to get a precise answer to the question of the height of the raised floor, it is recommended to perform a CFD simulation of the air flows, to calculate several options and to choose the most appropriate.
What is CFD simulation?
CFD is an acronym for Computational Fluid Dynamics.
Using specialized software, the user creates a three-dimensional model of an object, imposes certain boundary conditions, selects models representing physical phenomena occurring in gaseous and liquid media (heat transfer, media flow, thermal conductivity, radiation, convection, etc.), selects a calculation method and performs calculations… Based on the calculation results obtained, the user evaluates, if necessary modifies the computer model, and performs the calculations again. ), selects a calculation method and performs calculations… On the basis of the calculation results obtained, the user evaluates, if necessary, modifies the computer model, and performs the calculations again. The purpose of modeling is to write the physical phenomena as well as possible and then to find an adapted and satisfactory solution to the design problems that may be encountered.
Simulation results are used when making design decisions, to further improve the created plant model, identify bottlenecks in the operating plant and optimize the operating system.
Using CFD simulation in data centers
Unfortunately, higher operating temperatures can reduce the reaction time in case of a rapid increase in temperature due to a failure of the cooling unit. A data center containing servers operating at higher temperatures may experience instantly of hardware failures simultaneous. Recent ASHRAE regulations emphasize the importance of proactive monitoring of environmental temperatures inside server rooms.Data centers are ideal objects for IT modeling, because it is impossible to create a prototype or physical model of a data center. And without creating a data center model, it is impossible to predict with sufficient accuracy how the air conditioning system will perform in a real operating facility, how the air conditioning system will behave when the load changes, how the temperature will change in a row of server racks and along the height of each rack.
When designing an air conditioning system in a data center, a large number of parameters must be taken into account. We will give here some of them:
- The size and the volume of the room;
- Placement of telecommunications cabinets and racks in the server room;
- Floor height raised; direction, volume and speed of movement of cold air currents;
- Location of air-conditioning equipment;
- The types of fans used and the direction of the air flow,
- Consideration of obstacles to the flow of cold air;
- The type of raised access floor slabs used and the geometry of the outlets.
When designing an air-conditioning system without using CFD analysis, most of these parameters are not properly taken into account or are overdimensioned. Indeed, the real influence on the distribution of temperature and humidity in the data center room according to the studied parameter cannot be reliably estimated without precise computer simulation.
CFD software for data centers
There are a large number of programs on the market that solve various problems related to the simulation of flows of liquids and gases. These programs include the following: ANSYS, Autodesk CFD, Xflow, Open Foam, Phoenics, Flow Vent,STAR-CD , FASTEST-3, Flow Vision, Tile Flow, Sigma6room, Gas Dynamics Tool… However, not all thermodynamic flow simulation programs have ready-to-use modules and libraries of integrated elements, taking into account the specificities of data centers.
Software such as Tile Flow and Sigma 6 have built-in modules, programs and libraries to simulate airflow in the data center. For engineers not used to working with CFD modeling programs, it makes sense to consider purchasing this type of software, which already contains ready-to-use models for calculating air flows in the data center, there are libraries of equipment (e.g. fans, pumps, air conditioning units). In all cases, what makes the quality of the study is the experience level of the engineer in charge of the simulation. CFD engineering must be done by specialists.
Data center CFD modeling design steps
Recovering study hypotheses
Before carrying out the modeling process for an existing data center, it is necessary to carry out a complete and accurate study of the object: measure air flow velocity, measure pressure, take temperature measurements, determine air circulation channels and detect obstacles and possible locations of air leaks. In other words, the task of examining an existing object itself is quite laborious but, nevertheless, extremely useful. Because in the data collection process, bottlenecks are identified. To solve the problem of creating a model for a new data center, it is necessary to collect initial data from the space and validate assumptions about the technologies and devices used.
3D data center modeling
Next, create a geometric model of the data center (or digital twin) and the elements that make up the data center. A 3D model of an object is created using CAD programs, then the data is exported to the CFD simulation module.
Mesh creation
Next, the resolution model is created. This step is performed in programs using integrated mesh generation software modules or by using separate software products. The accuracy, the convergence and the speed of calculation depend on the mesh . The quality of the results obtained depends directly on the quality of the mesh (fineness, mesh adaptation…). After the mesh construction step, the user has to check the quality of the constructed mesh by different parameters (asymmetry of the elements, ratio height / width).
Boundary conditions & simulations
Boundary conditions are entered into the program and models are selected according to the assumptions, then a calculation is performed, which may converge or diverge (i.e. not have a correct solution) according to the various parameters above.
After convergence, the results of the calculations can be processed by special programs and displayed in graphical, tabular or evenanimated form, clearly demonstrating the changes in the physical parameters. For data centers, a visual representation of calculated data is usually used in the form of temperature distribution over the area of the computer room and the height of the server racks.
The engineer then analyzes the calculated results and, if necessary, modifies the object models and performs the calculations again.
What impact does this have on design?
Modern design tools enable CFD engineers to communicate with the various trades, so as to simply explain the phenomena at the root of problems, and then propose propose solutions which can then be validated collectively.
Summary
CFD programs can simulate the flow of liquids and gases, as well as other physical phenomena associated with this process, such as heat transfer. Thermodynamic modeling offers great opportunities to analyze the flow of liquids and gases, which allows to design at a high professional level new systems, equipment or tooptimize the operation of existing systems.
Without the use of CFD modeling, it is impossible to obtain precise answers to such fundamental questions as the distribution of temperatures and humidity along the cold aisles and the height of the server rack room, as a function of :
- of the thermal load;
- of the installation sites of the air conditioning units;
- heat transfer fluid and refrigerant temperatures ;
- the height of the raised floor;
- the distribution of broadcast schedules;
- fan type and other parameters.
