The previous answers are correct about strength and stress concentrations, but airplanes and boats have a different concern about what might break.
On a boat, the steel hull is very strong. The concern is that when a wave slams into the window the weaker glass might brake. On a pressurized airplane the fuselage is thin aluminum (but this is changing to composite materials). Aluminum is more susceptible to fatigue cracks than steel. The concern with airplanes is not that the window will break, but that sharp corners in the window frame will lead to fatigue cracks in the fuselage of the airplane.
You know how if you’re opening a snack bag, it’s much easier to open ones with a little notch to tear from?
Well if you’re tearing apart an airplane, it’s much easier to do that if there’s some sort of notch to tear from, like the corner of a rectangle cut in the skin.
If you don’t use any sharp corners, that makes the whole skin much less likely to develop “tears” from all the forces acting on it in flight.
The de Havilland comet was a 50s era early jet airliner with square windows, it was seen as futuristic at the time, but they were abandoned after redesign work following three mid air breakups of the aircraft resulting from metal fatigue or other structural failure (not bombed or shot down)
Basically square holes cause concentrations of stress at the corners.
“Rounded corners are designed to help evenly distribute the pressure exerted on the window, reducing the likelihood of a window cracking under changing air pressure,” It is mainly due to structural integrity. The ocean puts a lot of pressure on the body of the ship and square windows are more susceptible to stress. Rectangular or square windows tend to be weaker in some spots as compared to others. A round design is logically tougher and easier to reinforce. Name of the round window in a ship. A porthole, sometimes called bull’s-eye window or bull’s-eye, is a generally circular window used on the hull of ships to admit light and air
Latest Answers