Python logging change level through context manager
My standard logging setup is logger=logging.getLogger(__package__) in my main runner file and .getLogger(__name__) for all other files.
I wanted to temporarily change the loglevel of a specific logger of a library. Logical thing is to use a context manager, and such things exist:
I liked the second one, but what I wanted is to change the loglevel of another logger.
Usage:
# inside somelib.data...
liblogger = logging.getLogger(__name__)
logger.info("Stuff")
liblogger.info("Stuff from the lib")
with LoggingContext(
"somelib.data",
level=logging.ERROR
):
# very deep inside somelib.data...
liblogger.warning("Useless warning")
liblogger.warning("Not useless warning")
logger.info("Stuff")
Idea:
- While inside the context, the loglevel of the logger used inside the library gets set to ERROR
- I see only ERRORs from inside the library
- I don’t see their useless warnings that would be
logger.debug()s in my world
- Other loggers are unchanged
- On end of context everything goes back to normal
Second draft with operators!
But if I’m debugging I want these useless warnings!
After doing level=logging.ERROR if logger.level != logging.DEBUG else logging.getLogger('somelib_data').level oneliners I decided that I want the context manager to be flexible.
Ended up with this:
class LoggingContext:
"""Temporarily change the loglevel of a logger based on loglevels of
other loggers or arbitrary conditions."""
def __init__(
self,
logger_name: str,
level_true: int,
level_false: Optional[int] = None,
l1: Union[logging.Logger, int] = logger,
l2: Optional[int] = None,
comp_fn: Optional[Callable] = lambda x, y: True,
):
"""Temporarily change logging level of a logger, optionally dependent
on another logger's level.
:param logger_name: Change the level of a logger with this name
if None, the `level` new logger level will be used
:param callable_for_unchanged: if set, will be used to compare
main_logger_level to comparison logger level
and if True, will leave everything unchanged.
:param level_true: which loglevel to set in logger if condition is True
:param level_false: loglevel to set if condition is False
None means "don't change anything"
:param l1: main logger whose effective loglevel we'll use, or a loglevel
if None the global `logger` will be used
:param l2: loglevel to compare l1 with
if None will compare to the loglevel `level_true`
:param comp_fn: callable taking two params, loglevels/ints l1 and l2,
returning a boolean. Can be a lambda function or `operators` library
operators (eq,neq etc.)
If None will return True, ergo setting level_true always
"""
self.other_logger = logging.getLogger(logger_name)
# If it's a logger, get its effective level, if int - use that
main_level = (
l1.getEffectiveLevel() if isinstance(l1, logging.Logger) else l1
)
# Compare to l2 if it's there, otherwise to level_true
effective_comparison_level = l2 if l2 else level_true
# If callable is True, leave everything unchanged
comparison_result = comp_fn(main_level, effective_comparison_level)
# If we have no level_false, interpret it as "don't change anything"
if comparison_result:
self.level = level_true
else:
# 'None' here is a magic value "don't change anything"
self.level = level_false
logger.debug(
f"{logger_name=}, {l1=}, {l2=}, "
f"{level_true=}, {level_false=}, {comp_fn=}"
)
logger.debug(
f"{self.other_logger=}, {self.level=}, {main_level=}, "
f"{effective_comparison_level=}, {comparison_result=}"
)
if self.level is not None:
logger.debug(f"Changing {logger_name=} to loglevel {self.level}")
else:
logger.debug(f"Leaving {logger_name=} unchanged.")
def __enter__(self):
if self.level is None:
return None
self.old_level = self.other_logger.level
self.other_logger.setLevel(self.level)
def __exit__(self, et, ev, tb):
if self.level is None:
return None
else:
self.other_logger.setLevel(self.old_level)
This changes the idea completely and brings some VERY non-intuitive dynamics with default values, not sure yet if it’s worth doing it like that for the sake of brevity but we’ll see.
level_true,level_falseare levels to use based on conditionl1,l2are the two loglevels we comparecond_fnis a Callable/lambda/… that does the condition and returns a boolean.- Non-intuitive dynamics and default values. If omitted:
level_falsemeans “no change to status quo”l1takes the globallogger, which is probably a child of theloggerwe care about and inherits its effective loglevell2becomeslevel_true- For cases like “change loglevel to X only if X is more/less/equal than/to our
l1”
- For cases like “change loglevel to X only if X is more/less/equal than/to our
EXAMPLE SCENARIOS
- temporarily silence useless warnings of a library’s logger ‘other’:
with LoggingContext('other', logging.ERROR): - temporarily change loglevel of ‘other’, only if they’ll still be visible
to me afterwards (=level higher than current one):
with LoggingContext('other', logging.INFO, comp_fn=operators.le): - temporarily change loglevel of ‘other’ to shut it
up unless we’re in debug mode, in which case I want to see everything:
with LoggingContext('other', logging.ERROR, l2=logging.DEBUG, comp_fn=operators.eq): - if we’re at loglevel INFO or less, change ‘other’ to WARNING,
if not - otherwise change it to ERROR
from operators import le as less_or_equal with LoggingContext('other', level_true=logging.WARNING, level_false=logging.ERROR, l1=logger.level, # just as demo, it's implicit everywhere l2=logging.INFO, comp_fn=less_or_equal):`
Initially it was lambdas, but I kept wishing for “can I just pass <= as a function?” and lo and behold - yes, through the operator library!
Fazit
That was fun, and TIL about operators. In any case - another function for my small library of snippets.
Best of all, my favourite python blog has an article about the topic:The Unknown Features of Python’s Operator Module | Martin Heinz | Personal Website & Blog
Let’s see if I end up using this utility function more than once.
Bonus
Another similar-ish snippet I wrote once and still love. You get pretty progress bars only if you have enough elements in your sequence for it to make sense:
def _tqdm(list_like: Sequence, iflarge: bool = False, lsize: int = 100, **kwargs):
"""Use tqdm if it's on, optionally based on length of list.
Args:
list_like: thing to iterate.
iflarge (bool): If on, will use tqdm only for large lists
lsize (int): anything more than this is 'large'
**kwargs: get passed to tqdm as-is
"""
if USE_TQDM:
if not iflarge:
return tqdm(list_like, **kwargs)
else:
# Count size only if it doesn't mean iterating an iterator
if isinstance(list_like, Sequence) and len(list_like) > lsize:
return tqdm(list_like, **kwargs)
return list_like
Then, if the global USE_TQDM is true:
for x in _tqdm(sth)is a vanilla tqdmfor x in _tqdm(sth, True)becomes atqdmonly if we’re iterating through something larger than 100 elements._tqdm(sth, True, 50, desc="DOCS")tqdm on 50+ elements with a label (how cool is that?)
And on the same topic:
def log(msg) -> None:
"""Use loglevel.debug if tqdm is used, loglevel.info otherwise."""
if USE_TQDM:
logger.debug(msg)
else:
logger.info(msg)
logger.info() destroy tqdms, so - if we’re using TQDM, log it as logger.debug(). We’ll still see it on that loglevel if we want to (or maybe we’re logging it to a file, who knows).
TODO
- I think the RIGHT way to solve this would be a logging.Filter object. Later.
- I want a stable workflow that logs everything to a logfile but shows only a subset on screen. This means setting loglevel DEBUG, and adding a handler of loglevel INFO for stdout and a FileHandler of same DEBUG level for a file.