serhii.net

In the middle of the desert you can say anything you want

14 Sep 2022

Python logging change level through context manager

My standard logging setup is logger=logging.getLogger(__package__) in my main runner file and .getLogger(__name__) for all other files.

I wanted to temporarily change the loglevel of a specific logger of a library. Logical thing is to use a context manager, and such things exist:

I liked the second one, but what I wanted is to change the loglevel of another logger.

Usage:

# inside somelib.data...
liblogger = logging.getLogger(__name__)

logger.info("Stuff")
liblogger.info("Stuff from the lib")
with LoggingContext(
	"somelib.data",
	level=logging.ERROR
):
	# very deep inside somelib.data...
	liblogger.warning("Useless warning")

liblogger.warning("Not useless warning")
logger.info("Stuff")

Idea:

  • While inside the context, the loglevel of the logger used inside the library gets set to ERROR
    • I see only ERRORs from inside the library
    • I don’t see their useless warnings that would be logger.debug()s in my world
  • Other loggers are unchanged
  • On end of context everything goes back to normal

Second draft with operators!

But if I’m debugging I want these useless warnings!

After doing level=logging.ERROR if logger.level != logging.DEBUG else logging.getLogger('somelib_data').level oneliners I decided that I want the context manager to be flexible.

Ended up with this:

class LoggingContext:
    """Temporarily change the loglevel of a logger based on loglevels of
    other loggers or arbitrary conditions."""

    def __init__(
        self,
        logger_name: str,
        level_true: int,
        level_false: Optional[int] = None,
        l1: Union[logging.Logger, int] = logger,
        l2: Optional[int] = None,
        comp_fn: Optional[Callable] = lambda x, y: True,
    ):
        """Temporarily change logging level of a logger, optionally dependent
        on another logger's level.

        :param logger_name: Change the level of a logger with this name
            if None, the `level` new logger level will be used
        :param callable_for_unchanged: if set, will be used to compare
            main_logger_level to comparison logger level
            and if True, will leave everything unchanged.
        :param level_true: which loglevel to set in logger if condition is True
        :param level_false: loglevel to set if condition is False
            None means "don't change anything"
        :param l1: main logger whose effective loglevel we'll use, or a loglevel
            if None the global `logger` will be used
        :param l2: loglevel to compare l1 with
            if None will compare to the loglevel `level_true`
        :param comp_fn: callable taking two params, loglevels/ints l1 and l2,
            returning a boolean. Can be a lambda function or `operators` library
            operators (eq,neq etc.)
            If None will return True, ergo setting level_true always
        """
        self.other_logger = logging.getLogger(logger_name)

        # If it's a logger, get its effective level, if int - use that
        main_level = (
            l1.getEffectiveLevel() if isinstance(l1, logging.Logger) else l1
        )

        # Compare to l2 if it's there, otherwise to level_true
        effective_comparison_level = l2 if l2 else level_true

        # If callable is True, leave everything unchanged
        comparison_result = comp_fn(main_level, effective_comparison_level)

        # If we have no level_false, interpret it as "don't change anything"
        if comparison_result:
            self.level = level_true
        else:
            # 'None' here is a magic value "don't change anything"
            self.level = level_false

        logger.debug(
            f"{logger_name=}, {l1=}, {l2=}, "
            f"{level_true=}, {level_false=}, {comp_fn=}"
        )
        logger.debug(
            f"{self.other_logger=}, {self.level=}, {main_level=}, "
            f"{effective_comparison_level=}, {comparison_result=}"
        )

        if self.level is not None:
            logger.debug(f"Changing {logger_name=} to loglevel {self.level}")
        else:
            logger.debug(f"Leaving {logger_name=} unchanged.")

    def __enter__(self):
        if self.level is None:
            return None

        self.old_level = self.other_logger.level
        self.other_logger.setLevel(self.level)

    def __exit__(self, et, ev, tb):
        if self.level is None:
            return None
        else:
            self.other_logger.setLevel(self.old_level)

This changes the idea completely and brings some VERY non-intuitive dynamics with default values, not sure yet if it’s worth doing it like that for the sake of brevity but we’ll see.

  • level_true, level_false are levels to use based on condition
  • l1, l2 are the two loglevels we compare
  • cond_fn is a Callable/lambda/… that does the condition and returns a boolean.
  • Non-intuitive dynamics and default values. If omitted:
    • level_false means “no change to status quo”
    • l1 takes the global logger, which is probably a child of the logger we care about and inherits its effective loglevel
    • l2 becomes level_true
      • For cases like “change loglevel to X only if X is more/less/equal than/to our l1

EXAMPLE SCENARIOS

  • temporarily silence useless warnings of a library’s logger ‘other’:
    with LoggingContext('other', logging.ERROR):
  • temporarily change loglevel of ‘other’, only if they’ll still be visible to me afterwards (=level higher than current one):
    with LoggingContext('other', logging.INFO, comp_fn=operators.le):
  • temporarily change loglevel of ‘other’ to shut it up unless we’re in debug mode, in which case I want to see everything:
    with LoggingContext('other', logging.ERROR,
     l2=logging.DEBUG, comp_fn=operators.eq):
    
  • if we’re at loglevel INFO or less, change ‘other’ to WARNING, if not - otherwise change it to ERROR
    from operators import le as less_or_equal
    
    with LoggingContext('other', level_true=logging.WARNING,
    level_false=logging.ERROR,
    l1=logger.level,  # just as demo, it's implicit everywhere
    l2=logging.INFO, comp_fn=less_or_equal):`
    

Initially it was lambdas, but I kept wishing for “can I just pass <= as a function?” and lo and behold - yes, through the operator library!

Fazit

That was fun, and TIL about operators. In any case - another function for my small library of snippets.

Best of all, my favourite python blog has an article about the topic:The Unknown Features of Python’s Operator Module | Martin Heinz | Personal Website & Blog

Let’s see if I end up using this utility function more than once.

Bonus

Another similar-ish snippet I wrote once and still love. You get pretty progress bars only if you have enough elements in your sequence for it to make sense:

def _tqdm(list_like: Sequence, iflarge: bool = False, lsize: int = 100, **kwargs):
    """Use tqdm if it's on, optionally based on length of list.

    Args:
        list_like: thing to iterate.
        iflarge (bool): If on, will use tqdm only for large lists
        lsize (int): anything more than this is 'large'
        **kwargs: get passed to tqdm as-is
    """

    if USE_TQDM:
        if not iflarge:
            return tqdm(list_like, **kwargs)
        else:
            # Count size only if it doesn't mean iterating an iterator
            if isinstance(list_like, Sequence) and len(list_like) > lsize:
                return tqdm(list_like, **kwargs)

    return list_like

Then, if the global USE_TQDM is true:

  • for x in _tqdm(sth) is a vanilla tqdm
  • for x in _tqdm(sth, True) becomes a tqdm only if we’re iterating through something larger than 100 elements.
  • _tqdm(sth, True, 50, desc="DOCS") tqdm on 50+ elements with a label (how cool is that?)

And on the same topic:

def log(msg) -> None:
    """Use loglevel.debug if tqdm is used, loglevel.info otherwise."""
    if USE_TQDM:
        logger.debug(msg)
    else:
        logger.info(msg)

logger.info() destroy tqdms, so - if we’re using TQDM, log it as logger.debug(). We’ll still see it on that loglevel if we want to (or maybe we’re logging it to a file, who knows).

TODO

  • I think the RIGHT way to solve this would be a logging.Filter object. Later.
  • I want a stable workflow that logs everything to a logfile but shows only a subset on screen. This means setting loglevel DEBUG, and adding a handler of loglevel INFO for stdout and a FileHandler of same DEBUG level for a file.
Nel mezzo del deserto posso dire tutto quello che voglio.
comments powered by Disqus