The implementation of String.Join internally calculates the expected output length. If the calculated length is greater than max int32 (~2 billion characters), it will throw an "OutOfMemoryException" even though the system is not out of memory at that time .
It takes about 4GB to store 2 billion characters; since .NET stores strings in UTF-16 format, two bytes per character. However, .NET has an array memory limit of 2GB (unless on a 64-bit environment with <gcAllowVeryLargeObjects> enabled). As the code comment noted, this pre-check only catches the extreme case. The remaining of the method may still cause an OutOfMemory exception even if the pre-check passes.
For the record, .NET has an InsufficientMemoryException class for situations like this. However, this exception type was introduces in .NET 2.0. To stay backward compatible, the method remains throwing the OutOfMemoryException.
The following snippet comes from the official implementation of the method: