decorators — walk through a real logging or retry decorator

Why This Matters

During an interview, you're asked to design and implement a Python decorator that logs various aspects of function calls, including the function name, arguments, return value, execution time, and even the function source code. Additionally, discuss failure modes, edge cases, verification steps, and interview follow-ups. The decorator should also handle functions with no arguments, multiple calls, functions with exceptions, generator functions, coroutines, recursive function calls, specific function logging, excluding certain functions from being logged, logging the decorator itself and its arguments, and storing the logged information in a file instead of printing it.

Short Answer

To build a comprehensive decorator for logging function calls, first define an inner function (wrapper) to log the relevant information, then apply it as a decorator to the target function using the @ syntax. Here's a simple example:

def logger(func):
def wrapper(*args, **kwargs):
import time
start_time = time.time()
result = func(*args, **kwargs)
end_time = time.time()
print(f"{func.__name__} called with args: {args}, kwargs: {kwargs}, return value: {result}, execution time: {end_time - start_time}")
return result
return wrapper

Model Answer

Follow-Up Q And A

Q1: What happens if the decorated function doesn't take any arguments?

Answer: The logger decorator will still work as expected, but it won't print any arguments since there are none to log. To handle this case, you can modify the inner function to check for the presence of args and kwargs. Here's an updated version of the logger decorator:

def logger(func):
def wrapper(*args, **kwargs):
if args and kwargs:
start_time = time.time()
result = func(*args, **kwargs)
end_time = time.time()
print(f"{func.__name__} called with args: {args}, kwargs: {kwargs},\nReturn value: {result},\nExecution time: {end_time - start_time}")
return result
else:
print(f"{func.__name__} called with no arguments.")
return wrapper

Q2: Can we time the execution of a decorated function with multiple calls?

Answer: Yes, you can measure the total execution time of a decorated function across multiple calls. To do this, you'll need to maintain a variable outside the decorator's scope (e.g., global or class-level) to store the cumulative execution time and update it for each call. Here's an example using a global variable:

total_execution_time = 0

def logger(func):
def wrapper(*args, **kwargs):
nonlocal total_execution_time
start_time = time.time()
result = func(*args, **kwargs)
end_time = time.time()
print(f"{func.__name__} called with args: {args}, kwargs: {kwargs},\nReturn value: {result},\nExecution time: {end_time - start_time}")
total_execution_time += end_time - start_time
return result
return wrapper

Q3: What if we want to log more detailed information, like the function's source code?

Answer: To log the source code of a function, you can use Python's inspect module. You'll need to import it in your decorator and use its functions to access the source code. Here's an example:

import inspect

def logger(func):
def wrapper(*args, **kwargs):
src = inspect.getsource(func)
start_time = time.time()
result = func(*args, **kwargs)
end_time = time.time()
print(f"{func.__name__} called with args: {args}, kwargs: {kwargs},\nSource code:\n{src}\nReturn value: {result},\nExecution time: {end_time - start_time}")
return result
return wrapper

Q4: What if the decorated function raises an exception?

Answer: If a decorated function raises an exception, the log output will include the exception details along with the execution time. However, the original exception is still propagated to the caller. To prevent this, you can wrap the call to the decorated function in a try-except block and handle the exceptions within the decorator.

def logger(func):
def wrapper(*args, **kwargs):
try:
start_time = time.time()
result = func(*args, **kwargs)
end_time = time.time()
print(f"{func.__name__} called with args: {args}, kwargs: {kwargs},\nReturn value: {result},\nExecution time: {end_time - start_time}")
return result
except Exception as e:
print(f"{func.__name__} raised an exception: {e}")
return wrapper

Q5: What if the decorated function is a generator?

Answer: If the decorated function is a generator, you'll need to capture its yielded values and log them accordingly. Here's an example using a for loop to iterate through the generated values:

def logger(func):
def wrapper(*args, **kwargs):
start_time = time.time()
try:
result = []
for value in func(*args, **kwargs):
result.append(value)
end_time = time.time()
print(f"{func.__name__} called with args: {args}, kwargs: {kwargs},\nGenerated values:\n{result}\nReturn value: {result},\nExecution time: {end_time - start_time}")
except Exception as e:
print(f"{func.__name__} raised an exception: {e}")
return result
return wrapper

Q6: What if the decorated function is a coroutine?

Answer: If the decorated function is a coroutine, you'll need to use async/await syntax to properly log the execution time and yielded values. Here's an example using the asyncio library:

import asyncio

def logger(func):
async def wrapper(*args, **kwargs):
nonlocal start_time
start_time = time.monotonic()
try:
result = await func(*args, **kwargs)
except Exception as e:
print(f"{func.__name__} raised an exception: {e}")
end_time = time.monotonic()
print(f"{func.__name__} called with args: {args}, kwargs: {kwargs},\nReturn value: {result},\nExecution time: {end_time - start_time}")
return result
return wrapper

Q7: What if we want to log function calls recursively?

Answer: To log function calls recursively, you can modify the wrapper function to check whether it's already wrapping another function (i.e., if wrapper is not equal to func) and call itself recursively in that case. Here's an example:

def logger(func):
def wrapper(*args, **kwargs):
if wrapper != func:
return wrapper(*args, **kwargs)

import time
start_time = time.time()
result = func(*args, **kwargs)
end_time = time.time()
print(f"{func.__name__} called with args: {args}, kwargs: {kwargs}, return value: {result}, execution time: {end_time - start_time}")
return result
return wrapper

Q8: What if we want to log only specific functions or exclude certain functions from being logged?

Answer: To log only specific functions or exclude certain functions from being logged, you can modify the decorator to check the name of the decorated function against a list of allowed or disallowed names. Here's an example:

ALLOWED_FUNCTIONS = ["add", "multiply"]

def logger(func):
if func.__name__ not in ALLOWED_FUNCTIONS:
return func

def wrapper(*args, **kwargs):
import time
start_time = time.time()
result = func(*args, **kwargs)
end_time = time.time()
print(f"{func.__name__} called with args: {args}, kwargs: {kwargs},\nReturn value: {result},\nExecution time: {end_time - start_time}")
return result
return wrapper

Q9: What if we want to log the decorator itself and its arguments?

Answer: To log the decorator and its arguments, you can modify the wrapper function to print the decorator's name and arguments before logging the decorated function's information. Here's an example:

def logger(func):
def wrapper(*args, **kwargs):
import time
start_time = time.time()
print(f"Applying {logger.__name__} decorator to {func.__name__}")
result = func(*args, **kwargs)
end_time = time.time()
print(f"{func.__name__} called with args: {args}, kwargs: {kwargs},\nReturn value: {result},\nExecution time: {end_time - start_time}")
return result
return wrapper

Q10: What if we want to store the logged information in a file instead of printing it?

Answer: To store the logged information in a file, you can modify the wrapper function to open a file, write the relevant information, and close the file after each call. Here's an example using the open and write functions:

def log_to_file(filename):
def decorator(func):
def wrapper(*args, **kwargs):
with open(filename, "a") as f:
import time
start_time = time.time()
result = func(*args, **kwargs)
end_time = time.time()
f.write(f"{func.__name__} called with args: {args}, kwargs: {kwargs},\nReturn value: {result},\nExecution time: {end_time - start_time}\n")
return result
return wrapper
return decorator

Common Mistakes

• Forgetting to define the inner function (wrapper) or apply it as a decorator using @.
• Not updating the start_time variable before calling the decorated function.
• Returning the log output instead of the original result from the decorated function.
• Using incorrect Python syntax for defining decorators, such as missing the parentheses around the arguments in the wrapper function or forgetting to use the @ syntax.
• Failing to handle edge cases, like functions with no arguments, or not accounting for multiple calls when measuring execution time.

Edge Cases and Failure Modes

Q1: What if the decorated function's return value is a generator?

Answer: If the decorated function returns a generator, you'll need to handle it appropriately in the logger decorator. One way to do this is by capturing the generated values and logging them as part of the returned result. Here's an example:

def logger(func):
def wrapper(*args, **kwargs):
start_time = time.time()
try:
result = []
for value in func(*args, **kwargs):
result.append(value)
end_time = time.time()
print(f"{func.__name__} called with args: {args}, kwargs: {kwargs},\nGenerated values:\n{result}\nReturn value: {result},\nExecution time: {end_time - start_time}")
except Exception as e:
print(f"{func.__name__} raised an exception: {e}")
return result
return wrapper

Q2: What if the decorated function modifies its own code or other functions' codes (e.g., using globals() or locals())?

Answer: If the decorated function modifies its own code or other functions' codes, it can potentially affect the behavior of the decorator and lead to unpredictable results. To handle this case, you can create a copy of the original function's code before applying the decorator, then restore the original code after logging the execution. Here's an example using inspect for code extraction and restoration:

import inspect

def logger(func):
def wrapper(*args, **kwargs):
src = inspect.getsource(func)
original_code = func.__code__
func.__code__ = compile(src, "<string>", "exec")
start_time = time.time()
result = func(*args, **kwargs)
end_time = time.time()
print(f"{func.__name__} called with args: {args}, kwargs: {kwargs},\nReturn value: {result},\nExecution time: {end_time - start_time}")
func.__code__ = original_code
return result
return wrapper

Verification Steps

To verify the decorator's functionality, you can create test functions and call them with various arguments, check the log output for accuracy, and ensure that the execution time is correctly measured. You can also add assertions to the test cases to validate the return values of the decorated functions.

Here's an example using a simple test function and the logger decorator:

def add(a, b):
return a + b

@logger
def multiply(a, b):
return a * b

def test_logger():
assert add(2, 3) == 5
print("add called correctly.")
assert multiply(2, 3) == 6
print("multiply called correctly.")

By running the test function, you can verify that both functions are decorated correctly and their execution times are logged as expected.

In an interview scenario, you might be asked to discuss the limitations of this decorator, handle more complex cases, or propose improvements. Be prepared to think critically about your solution and adapt it to various use cases.

Q11: What if we want to log the number of calls for each function?

Answer: To log the number of calls for each function, you can create a global dictionary to store the count of calls for each decorated function and increment its value on each call. Here's an example:

calls = {}

def logger(func):
def wrapper(*args, **kwargs):
if func.__name__ not in calls:
calls[func.__name__] = 1
else:
calls[func.__name__] += 1
start_time = time.time()
result = func(*args, **kwargs)
end_time = time.time()
print(f"{func.__name__} called {calls[func.__name__]} times with args: {args}, kwargs: {kwargs},\nReturn value: {result},\nExecution time: {end_time - start_time}")
return result
return wrapper

Q12: What if we want to log the line number and file name of each call?

Answer: To log the line number and file name of each call, you can modify the logger decorator to use inspect.stack() to get the caller's frame information. Here's an example:

import inspect

def logger(func):
def wrapper(*args, **kwargs):
frames = inspect.stack()[1:]
file_name = frames[0].filename
line_number = frames[0].lineno
func_name = func.__name__
start_time = time.time()
result = func(*args, **kwargs)
end_time = time.time()
print(f"{func_name} called at line {line_number} in file {file_name},\nReturn value: {result},\nExecution time: {end_time - start_time}")
return result
return wrapper

Interview Follow-Ups

During an interview, you might be asked follow-up questions about your solution, such as:

1. Can you optimize the decorator to reduce its overhead?
2. How would you handle a situation where the function being decorated is not defined at the time of decoration (e.g., in a separate module)?
3. What if we want to log additional information, like the line number and file name of each call?
4. Can you explain how the decorator works under the hood, especially when it comes to function calls and returning values?
5. How would you handle cases where the decorated function modifies its own code or other functions' codes (e.g., using globals() or locals())?
6. What are some potential performance issues with this decorator, and how can they be addressed?
7. Can you discuss any security concerns related to logging sensitive information, such as passwords or API keys?
8. How would you approach debugging a complex system that uses multiple decorators and potentially hundreds of decorated functions?