Customize logging in django

1 Add logging to the django app

Best logging tut
DjangoCon 2019 talk about logging
Find all of the loggers here. Get to know to root logger - "django", then db logger - "django.db.backends" and similar.
Example how I added logging in quotes app

Nobody is paying us to log, but it is needed, just as tests.

FIND and REPLACE all your print() with logger.info() or logger.debu().

Whenever you have shippable code, don't remove the debug logs.. no developer will run your code with debug logs on. And if there is a need for that - they will be glad that there are some debug logs in the code, that can act as a reminder and a great help to debug.

You can log your own log messages or log django's log messages as well. These are two different things.

Loggers - what to log

Handlers - how to log those logged logs

Formatters - how to display the logged logs

For example here is how to log out to a file your own log message. Add the logging configuration below to your `settings.py` file:

First let's make sure the logging directory exists:

# LOGGING START

LOGS_DIR = os.path.join(BASE_DIR, "logs")

if not os.path.exists(LOGS_DIR):
    os.makedirs(LOGS_DIR)

LOGGING = {
    "version": 1,
    "disable_existing_loggers": False,
    "handlers": {
        "file_handler": {
            "class": "logging.FileHandler",
            "filename": os.path.join(LOGS_DIR, "logs.log"),
            "level": "DEBUG",
            "formatter": "verbose",
        },
    },
    "loggers": {
        # unmapped aka root logger, catch logs from ALL modules (files)
        # this would show all of my written logs, like logger.error("hello!")
        "": {
            "level": "DEBUG",
            "handlers": ["file_handler"],
        }
    },
    "formatters": {
        "simple": {
            "format": "{asctime}: {levelname} {message}",
            "style": "{",
        },
        "verbose": {
            "format": "{asctime}: {levelname} - {name} {module}.py (line {lineno:d}). {message}",
            "style": "{",
        },
    },
}

Now for test the above logger, see if anything is written to `debug_up.log` file, let's add this anywhere in our project:

import logging
logger = logging.getLogger(__name__)
logger.debug("hello!")
logger.info("hello!")
logger.warning("hello!")
logger.error("hello!")
logger.critical("hello!")

You will see this in the terminal:

2024-05-01 08:55:44,301: DEBUG - apps.authors.models models.py (line 20). hello!
2024-05-01 08:55:44,302: INFO - apps.authors.models models.py (line 21). hello!
2024-05-01 08:55:44,303: WARNING - apps.authors.models models.py (line 22). hello!
2024-05-01 08:55:44,304: ERROR - apps.authors.models models.py (line 23). hello!
2024-05-01 08:55:44,305: CRITICAL - apps.authors.models models.py (line 24). hello!

Now if you also want to display the same logs in the console(terminal), do this:

LOGGING = {
    "version": 1,
    "disable_existing_loggers": False,
    "handlers": {
        "file_handler": {
            "class": "logging.FileHandler",
            "filename": os.path.join(LOGS_DIR, "logs.log"),
            "level": "DEBUG",
            "formatter": "verbose",
        },
        "console_handler": { # new
            "class": "logging.StreamHandler", # new
            "level": "DEBUG", # new
            "formatter": "simple", # new
        },
    },
    "loggers": {
        # unmapped aka root logger, catch logs from ALL modules (files)
        # this would show all of my written logs, like logger.info("Index view accessed by user: %s", request.user.username)

        "": {
            "level": "DEBUG",
            "handlers": [
                "file_handler",
                "console_handler", # new
            ],
        },
    },
    "formatters": {
        "simple": {
            "format": "{asctime}: {levelname} {message}",
            "style": "{",
        },
        "verbose": {
            "format": "{asctime}: {levelname} - {name} {module}.py (line {lineno:d}). {message}",
            "style": "{",
        },
    },
}

If I wanted to see django's logs, similar to debug toolbar, I would have to play with the loggers, instead of "", I could for example write "django", then create another file for those logs. Have not played much, but it should work.

Also be aware of logger namespacing, as well as logger hierarchies and propagation.

It also might be a good idea to store the debug levels, filenames and similar info to environment variables.

Can also later look into "log aggregation", store your logs and visualize in "prometheus" or "better stack"

It is also possible to add colors to logging, like so (have to `pip install colorlog` first). Add it to formatters, then use as a regular formatter in the console:

"colored": {
    "()": "colorlog.ColoredFormatter",
    "format": "%(log_color)s %(asctime)s: [%(levelname)s] - %(name)s %(module)s.py (line %(lineno)s). :: %(message)s",
},

If you want "rotating" logs, can add such configuration:

"handlers": {
    "rotating_file_handler": {
        "class": "logging.handlers.RotatingFileHandler",
            "filename": os.path.join(LOGS_DIR, "logs.log"),
        "mode": "a",
        "encoding": "utf-8",
        "formatter": "verbose",
        "backupCount": 5,
        "maxBytes": 10485760,
    },
}

If you want to see the SQL statements of each request, can add such logger:

## catch all the SQL that is generated by django's ORM
"django.db.backends": {
    "level": "DEBUG",
    "handlers": [
        "console_handler",
        "rotating_file_handler",
    ],
},

If you want to see django's logs, all logs, create such logger:

# catch logs from django (make this DEBUG to get loads of info)
"django": {
    "level": "INFO",
    "handlers": [
        "console_handler",
        "rotating_file_handler",
    ],
    "propagate": False,
},