The Basics of x86
x86 is one of the most common computer architectures in use today. Many personal computers and devices use either x86 or ARM CPUS. But what does that mean? What is the difference?
The Instruction Set
As you might have heard before, the way code is run at the lowest level is machine code. Machine code is made up of individual bytes known as opcodes. Each opcode is followed by one to two arguments. In essence, an opcode is just a function call. The set of all the opcodes a machine has is called its instruction set. x86 has a very big instruction set, comprised of thousands of opcodes. ARM is what is called a RISC, or reduced instruction set computer. That means its instruction set is smaller than normal: only about 300 instructions. That is why it is used for embedded devices.
Registers, Math, and the Stack
But what do all these opcodes actually do? Some of the most common ones are for manipulating registers and the stack. A register similar to a variable: it just stores a value, and you can manipulate it. There are many registers in x86. First, there are the general purpose registers:
esi, as well as
r9. You can store whatever data you want in these registers, and no one will complain. You might wonder what the names mean. The answer is that they indicate the size in bits of the register. If a register is named
<x>l, it is 8 bits. If it is named
<x>x, it is 16 bits. If it is named
e<x>x, it is 32 bits. Finally, if it is named
r<x>x, it is 64 bits. x86 also has a stack. You have probably heard of a stack: you can push and pop to it, but only pop the top item. The stack is handled by two registers:
esp. The stack on x86 grows downward: when you push an item, the top of the stack moves down one address. The stack is just a region of memory. The registers are used to manipulate it:
ebp points to the bottom of the stack, and
esp to the top. The stack is how C stores variables: you might have one variable at
esp - 8, another at
esp - 16, and another at
esp. The stack is handled by two instructions:
pop. The are pretty self-explanatory:
push pushes an item, and
pop pops it.
OSes and the Boot Process
Well that's all well and good, but what actually happens on boot up? If I turn on my computer, what does it do? How does it load my OS? What even is my OS? All those questions are covered by something called modes and the boot process. We will answer the last question first: an OS is made of a couple of parts: the bootloader and the kernel. The bootloader is what it sounds like: it is booted, and it loads the rest of the OS. The rest of the OS is called the kernel, and it is in charge of pretty much everything: file system, drivers, userspace and everything. On most Linux systems, the bootloader is GRUB or Syslinux. Bootloaders are a complex topic, and I won't go into them here. Now we will get to the actual boot process: how does the computer load the bootloader? When the computer turns on, it loads something called the BIOS, or Basic Input Output System. The BIOS loads 1 sector, or 512 byte group, from the boot disk to the address 0x7c00, and then jumps there. This is why bootloaders are necessary: the BIOS only reads 1 sector, so it is necessary to load more to have a legitimate OS. Most bootloaders do not do this immediately, though: they will load a second stage bootloader, which will load the kernel.
I/O and Interrupts
But how does the BIOS load the bootsector? It doesn't just automatically: it has to use what is called an I/O port or MMIO to interact with the hard disk. An I/O port is just like a network port, but on the computer's hardware: You can read and write to it, and each port has its own particular function. There is one for reading the mouse, another for the hard drive, and another for the graphics card. MMIO, or Memory Mapped Input Output, is another way I/O is done. The way it works is that the device will specify an address for data to be written to it, and read data from that memory address. For example, a VGA card will read data from the address
0xb8000 and write it to the screen. However, most things do not interface with the hardware directly like the BIOS: there are so many types of devices that to cover each one would use up the whole boot sector! Instead, an OS will use what is called a BIOS interrupt. These are done with the
int instruction. An interrupt basically tells the computer that the OS wants to do something, and the computer will find the code the BIOS to set up to do that. For example, the BIOS sets up interrupt
0x13 to perform hard disk services. Thus, to interface with the hard drive, I would simply call
int 0x13, and the computer would do that I wanted it to for me.
Modes and Memory
Well, BIOS interrupts are great, but they come at a price. When the computer boots up, it is in what is called Real mode or 16-bit mode: You have access to every aspect of the computer with no protections, and you can use BIOS interrupts. The problem is with memory access. The name 16-bit mode comes from the fact that you can only access 16-bit addresses: 0 to 65536. As you might expect, this is a problem: what if an app needs more than 65536 bytes of RAM? The solution is called Protected mode pr 32-bit mode. In Protected mode, you have access to 32-bit addresses, letting you read and write 4 GiB of RAM. You can also set privileges on it to only allow certain processes to access it. The price of this is that you cannot use BIOS interrupts anymore. Many older operating systems like DOS run in Real mode, but modern ones run in Protected mode or Long mode, which allows you to access 64-bit addresses.
Segmentation, Paging, and the GDT
Why is it that Real mode OSes can only use 16-bit addresses? The reason is that they use something called segmentation to access memory. Each area of memory is divided into a segment. You then access offsets within the segment. For example, you might have the segment
0x1000 and the offset
0x1234, giving you the address
0x1000:0x1234. To convert this to a physical address, we shift the segment left 4 bits and add the offset to that. By applying this, we get
(0x1000 << 4) + 0x1234 = 0x10000 + 0x1234 = 0x11234. The computer handles segments using segment registers:
gs. These are, respectively: the code segment, where the program runs, the data segment, where data is stored, the stack segment, for the stack, the extended segment, for user use, and two useless segments. Protected mode does things differently. Because of the size of a paging physical address, we can only access addresses that will fit in a segment:offset address. When you enter protected mode, you must set up something called a GDT, or Global Descriptor Table. This is where you define how you want to access memory: for example, you could set up your OS to only use addresses
0x90000. Once in protected mode, you do not use segmentation anymore: you must set up something called paging. In paging, you use what is called virtual memory. This allows you to have each process think that it has access to all 4 GiB of memory and that it runs at
0x0000, even if it does not. In paging, virtual and physical memory is divided into pages. Each page in virtual memory is mapped to a page in physical memory. A list of all the pages and their mappings is stored in a page table, which is in turn stored in a page directory where the computer can access it.
A Final Note: Emulators
If you want to write you own OS for x86, you will want to learn more about assembly language, the human readable form of machine code, and C. Then, you will need to write a bootloader and kernel, or set up a kernel to boot with GRUB or Syslinux. Finally, you will have a bootable image, which is basically just a file containing you OS as machine code. What do you do know? You need to know if you OS works! You could write the image to a flash drive, reboot your computer, and boot from the flash drive. But this is messy and wastes time. Instead, most OS developers use an emulator for testing. An emulator is simply a virtual machine that runs you OS: it emulates the x86 instruction set, so your OS thinks it is running on real hardware, when in reality it is just running as an app. Two good emulators are QEMU and VirtualBox. VirtualBox is geared more towards those who want to run other OSes on their machine, say run Linux without installing it. It is easiest to use QEMU to test your OS in development.
This concludes my tutorial on x86. If you liked it, be sure to check back later, as I plan to write other tutorials on Python and perhaps even writing a basic OS. Thanks for reading!
@sugarfi lol 69. I saw a post on SO the other day now that I think about it that was just about reducing the amount of space raw binary would take up when encoded. The problem was everything else was horribly slow, so I think eventually they just went with base64.
I think if I can figure out base 32....
@sugarfi @Highwayman QEMU works on replit. You just need to manually copy the missing files iirc. Also, @CSharpIsGud already made replos https://repl.it/talk/share/ReplOS-A-REAL-Operating-System-on-replit/30207
You may want to reword
Why is it that Real mode OSes can only use 16-bit addresses? The reason is that they use something called segmentation to access memory. - As segmentation lets you use addresses higher than 65536 its just that you have to go through segments instead of directly accessing an address