The first computers took instructions via [punch cards](https://en.m.wikipedia.org/wiki/Punched_card). So you’d literally load a huge stack of cards into the system and it would read the code.
Once it was in memory you could write that out to tape, and then subsequent machines would load the operating system from tape into memory.
Latest Answers