Understanding Process Isolation in Windows

Let's break down how two processes communicate in Windows, what process isolation means, and how APIs like WriteProcessMemory and VirtualAllocEx manage to access another process's memory — without breaking the rules.

Two Processes Can Communicate#

In Windows, two processes can communicate with each other. Even though each process has its own private virtual address space, they can still exchange data using kernel-mediated mechanisms such as shared memory, named pipes, sockets, or RPC (Remote Procedure Calls).

These mechanisms are controlled and safe, as they are managed by the kernel.

Then what Is Process Isolation ?#

Process Isolation means that one process cannot directly access another process's memory in user space. (note : cannot directly access )

Each process runs in its own private memory area, ensuring that:

• One faulty or malicious process can't modify another process's memory.
• The operating system remains stable and secure.

However, kernel-mediated mechanisms allow processes to exchange data safely — but only through controlled access paths, not direct memory access.

So yes, processes can communicate, but not directly in user space. Instead, they do it indirectly through the kernel.

How Do APIs Like WriteProcessMemory and VirtualAllocEx Work ?#

APIs like WriteProcessMemory and VirtualAllocEx can access another process's memory, but only if they have a valid process handle with the right permissions, such as PROCESS_VM_WRITE or PROCESS_VM_OPERATION.

Here's what really happens behind the scenes.

The whole WriteProcessMemory flow at a glance — Process A never reaches into Process B directly. The kernel reads the bytes from A's buffer and writes them into B's target address, the only legal path between two isolated processes. Follow the moving data block through the steps below.

Step-by-Step: How the Kernel Handles It#

1. Memory Layout of Each Process#

Process A has its own user space and a shared kernel space.

Process B also has its own user space and the same shared kernel space.

The kernel space is shared across all processes but is protected — only the kernel itself can access it. User-mode code in Process A or B cannot directly touch kernel memory.

2. Process Isolation Rules#

• Process A cannot directly read or write Process B's user space.
• The kernel enforces this isolation using page tables and privilege checks.
• User mode runs with restricted access; kernel mode runs with full privilege.

3. Getting a Handle (OpenProcess)#

When Process A calls OpenProcess(PROCESS_VM_WRITE, FALSE, B_PID), it is asking the kernel, "Can I get permission to write to Process B's memory?"

The kernel checks whether the caller (Process A) has the required permissions and whether it is allowed by security tokens and access rights.

If the check passes, the kernel returns a handle — a kind of permission ticket. It doesn't give direct access to memory; it's just a reference managed by the kernel.

4. Writing to the Other Process (WriteProcessMemory)#

Next, Process A uses that handle to call WriteProcessMemory(hProcessB, targetAddress, buffer, size, NULL).

• The call starts in user space (inside kernel32.dll) and goes through ntdll.dll, which triggers the system call NtWriteVirtualMemory
• The CPU switches from user mode to kernel mode — execution enters the Windows kernel (ntoskrnl.exe).
• The kernel checks whether the handle is valid, whether Process A has the required permissions, and whether the target address is within Process B's user space.
• The kernel temporarily maps Process B's target memory pages into its own kernel address space. Then, ntoskrnl.exe copies the data from Process A's buffer (in A's user space) into Process B's target address (in B's user space).

The kernel can do this because it runs in ring 0, the most privileged mode of the CPU. After the copy, the mapping is released — meaning no process holds a permanent reference to the other's memory.

5. What Actually Gets Modified#

The destination is still Process B's user-space memory.

But the operation is executed by the kernel, not directly by Process A.

So yes, the memory being modified belongs to Process B's user space, but it's done through kernel code, not by Process A's user-mode instructions.

In Summary#

• Process isolation ensures one process cannot directly access another's memory.
• Communication between processes happens through kernel-mediated mechanisms like pipes, shared memory, and sockets.
• APIs like WriteProcessMemory don't break isolation — they work with kernel permission, using temporary page mappings handled by ntoskrnl.exe.
• The kernel acts as the bridge, performing controlled memory operations between isolated processes.