2025-12-149 min read

Child Process Module (Python Programming)

Learn Child Process Module (Python Programming) step by step with clear examples and exercises.

Why This Matters

The Child Process Module is a crucial aspect of Python programming that enables developers to create and manage multiple processes concurrently. This module allows for more complex applications or scripts, improving efficiency in long-running tasks, parallel processing, and system administration tasks. By understanding the child process module, you can write more versatile and powerful scripts that take full advantage of your computer's resources.

Prerequisites

To fully understand this guide, you should have a good grasp of Python programming basics such as variables, functions, loops, and error handling. Familiarity with multiprocessing and threading is beneficial but not required. It's recommended that you have experience working with the command line and understanding basic shell commands.

Core Concept

The Child Process Module in Python provides an interface to create child processes using the subprocess module. The primary functions are Popen(), run(), and check_call().

Popen()

The Popen() function starts a new process with its own separate environment, returning a Popen object that can be used to communicate with the child process, control its lifecycle, and retrieve its output.

import subprocess

child_process = subprocess.Popen(["ls", "-l"])

In this example, we create a new child process that runs the ls -l command, listing all files in the current directory.

run() and check_call()

The run() function is similar to Popen(), but it captures the output of the command as a string and returns it. The check_call() function runs a command and checks if it exits successfully; otherwise, it raises an exception.

import subprocess

output = subprocess.run(["ls", "-l"], capture_output=True)
print(output.stdout.decode())

try:
subprocess.check_call(["command_that_doesnt_exist"])
except subprocess.CalledProcessError as e:
print("Command failed with error:", e.returncode)

In the first example, we run the ls -l command and capture its output as a string. In the second example, we attempt to execute a non-existent command, causing an exception to be raised if the command fails.

Communication between Parent and Child Processes

Communication between parent and child processes can be achieved using pipes, stdin/stdout/stderr, or a combination of both. Pipes allow you to send data from one process to another through a pipe object.

import subprocess

parent_input = "Hello, Child!"
child_output = ""

Create a pipe for communication between parent and child processes

pipe = subprocess.Pipeline(args=["cat", "-"], stdin=subprocess.PIPE, stdout=subprocess.PIPE)

Start the child process and send input through the pipe

child_process = pipe.run_async(input=parent_input.encode())

Read output from the pipe until there is no more data

while True:

line = pipe.recv(1024)

if not line:

break

child_output += line.decode()

print("Child output:", child_output)


In this example, we create a pipe and start a `cat` command that reads from stdin (`-`) and writes to stdout. We send the string "Hello, Child!" through the pipe and read the output until there is no more data.

### Redirecting Input/Output Streams

You can redirect input/output streams using the `stdin`, `stdout`, and `stderr` arguments when creating a child process. For example:

child_process = subprocess.Popen(["ls", "-l"], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

output = child_process.communicate()[0].decode()

print("Output:", output)


In this example, we redirect the standard output of the `ls -l` command to a pipe and capture its output as a string. The standard error stream is set to `STDOUT`, meaning it will be written to the same place as the standard output (the console in this case).

### Waiting for Child Processes

To wait for child processes to complete, you can use the `wait()` method of the `Popen` object. This blocks the parent process until the child process terminates.

child_process = subprocess.Popen(["ls", "-l"])

child_process.wait()

print("Child process completed.")


In this example, we create a new child process and wait for it to complete before continuing with the parent process.

Worked Example

Let's create a simple example where we run multiple instances of a script that generates prime numbers concurrently using child processes.

import subprocess
import os

def generate_primes(start, end):
primes = []
for num in range(start, end + 1):
if is_prime(num):
primes.append(num)
return primes

def is_prime(n):
if n <= 1:
return False
for i in range(2, int(n ** 0.5) + 1):
if n % i == 0:
return False
return True

def run_primes_generator(start, end, num_processes):
processes = []

for _ in range(num_processes):
process = subprocess.Popen(["python", "generate_primes.py", str(start), str(end)], stdout=subprocess.PIPE)
processes.append(process)

results = []
for process in processes:
result = process.communicate()[0].decode().splitlines()
results.extend(result)

primes = list(map(int, results))
return sorted(primes)

def main():
start = 1
end = 100
num_processes = 4

primes = run_primes_generator(start, end, num_processes)
print("Generated primes:", primes)

if __name__ == "__main__":
main()

In this example, we define a generate_primes() function that generates prime numbers within a specified range. We also create an is_prime() helper function to check if a number is prime. The run_primes_generator() function spawns multiple child processes that run the generate_primes() function concurrently, collects their output, and sorts the generated primes.

Common Mistakes

Not capturing output: When using Popen(), it's essential to capture the output of child processes if needed. You can do this by setting the stdout and/or stderr arguments to a pipe or file object.
Ignoring return codes: The check_call() function raises an exception when the child process exits with a non-zero status code, indicating an error occurred. Always handle these exceptions appropriately.
Incorrect handling of input/output streams: When communicating between parent and child processes using pipes or stdin/stdout/stderr, ensure that you properly read from and write to the appropriate streams in both the parent and child processes.
Not terminating child processes: If a child process is not terminated explicitly, it may continue running even after the parent process has finished, causing resource leaks. Always call process.terminate() or process.kill() when necessary.
Misunderstanding the GIL (Global Interpreter Lock): The Global Interpreter Lock in Python can limit the performance benefits of concurrent processing. Be aware of its implications and consider using threading for CPU-bound tasks instead if possible.
Not handling exceptions properly: When working with child processes, it's important to handle exceptions appropriately to ensure that your program continues running smoothly even when errors occur.
Using the wrong function for the task: Choose the appropriate function (Popen(), run(), or check_call()) based on your needs and the nature of the command you're executing.
Not waiting for child processes to complete: If you don't wait for child processes to complete, the parent process may continue running before all child processes have finished, causing unexpected behavior.
Not handling signals correctly: When working with long-running child processes, it's important to handle signals such as SIGINT (Ctrl+C) and SIGTERM properly to ensure that your program can be gracefully terminated.
Not considering platform differences: Some operating systems may have different behaviors or limitations when it comes to child processes. Be aware of these differences and adjust your code accordingly if necessary.

Practice Questions

Write a script that runs the top command and captures its output every 5 seconds for 30 seconds, then saves the output to a file.
Modify the worked example to generate prime numbers in the range of 1 to 1000 using 8 child processes.
Write a script that runs ping command for multiple hosts concurrently and collects their response times.
Implement a simple web server using child processes that can handle multiple client requests simultaneously.
Create a script that uses child processes to download multiple files from the internet concurrently, saving them to separate directories.
Write a script that runs a command on multiple remote servers concurrently using ssh and collects their output.
Implement a script that uses child processes to perform a long-running calculation on a large dataset, splitting the data into smaller chunks for each process.
Create a script that uses child processes to convert a large image file into multiple smaller images, with each child process handling a different section of the image.
Write a script that uses child processes to perform a dictionary lookup on a large dataset, where each child process handles a different letter range in the alphabet.
Implement a script that uses child processes to perform a brute force attack on a simple encryption algorithm, with each child process trying different keys concurrently.

FAQ

Why use child processes instead of threads? Child processes have separate memory spaces, which makes them more suitable for CPU-bound tasks and I/O-bound tasks where thread safety is not a concern. Threads share the same memory space, which can lead to synchronization issues in certain scenarios.
Can I use child processes for parallelizing my existing Python script? Yes! You can replace blocking functions with Popen() calls to run time-consuming tasks concurrently using child processes.
How do I handle errors when using child processes? Use the check_call() function or check the return code of the child process to determine if an error occurred. You can also capture output to help debug issues.
What are some common pitfalls when working with child processes? Common pitfalls include forgetting to capture output, ignoring return codes, mismanaging input/output streams, and not terminating child processes properly. Be aware of these issues and handle them accordingly.
How can I pass arguments to a child process? You can pass arguments to a child process by including them in the list passed to Popen() or run(). For example: subprocess.Popen(["my_script", "arg1", "arg2"]).
Can I redirect input/output streams of a child process? Yes! You can redirect input/output streams using the stdin, stdout, and stderr arguments when creating a child process. For example: subprocess.Popen(["ls", "-l"], stdout=subprocess.PIPE).
How do I wait for multiple child processes to complete? To wait for multiple child processes to complete, you can use the wait() method of the Popen object or the waitpid() function. For example: child_processes = [p1, p2, p3]; while len(child_processes) > 0: pid = child_processes[0].poll(); if pid is not None: child_processes.pop(0); else: child_processes[0].wait().
How can I communicate between parent and child processes? You can use pipes, stdin/stdout/stderr, or a combination of both to communicate between parent and child processes. Pipes allow you to send data from one process to another through a pipe object.
Can I run commands on remote servers using child processes? Yes! You can use ssh to run commands on remote servers and capture their output using the Popen() function or run() function. For example: subprocess.Popen(["ssh", "user@remote_server", "command"]).
How do I handle signals (SIGINT, SIGTERM) when working with child processes? To handle signals, you can use the signal module to set up signal handlers for your parent and child processes.