There are many possible sources of drafts in a home. You are probably aware of how proper weatherstripping on doors and windows are an important way to reduce energy loss and stopping drafts. If not, take a look at our articles on weatherstripping doors and windows. Other sources can include drafts through wood floors, attic access hatches, chimneys and any place in the exterior of your home where a pipe, flue, duct or other service passes through it.
However, on the inside of your home drafts can come from electrical outlets and switches. A hole was made through the drywall for an electrical box. There is often a gap around the box and the box itself is not air tight. Most interior walls are not insulated and so air can flow around in them easily. Even exterior walls with insulation sometimes have open pockets that allow airflow. The air gets into the walls because there are plenty of openings in the floor under your house or in the attic. A breeze blows outside and it finds its way through the walls and into your room.
A simple technique to stop these drafts is to remove the cover plate, install a small soft foam pad over the switch or outlet and replace the cover plate. These foam pads (aka gaskets) are made for the purpose and are available at most home centers. The pads have punch outs to fit snuggly around switches and outlets and they are a little larger than the electrical box, so they cover the gap.
While you might be doubtful about the difference this can make, the task is so easy to do and so inexpensive, it can hardly hurt to try. However, you can prove it to yourself. Wait for a chilly day, especially when there is a breeze outside. Light a candle and hold the flame near several switches and outlets around your home. Unless your home is already pretty tightly sealed up, you will very likely see your flame spluttering about.