You’re right that the fuel pump relay is located in a fuse box, but the 2004 Mazda 6’s fuel pump relay is located in the engine compartment fuse box. This is on the driver’s side of the engine bay just to the right of the battery. The fuse you need should be located in slot 18, but to be sure of the number, you should check the diagram on the box cover and/or your owner’s manual.
A faulty fuel pump relay can often cause your engine to stall—and could even lead to a complete breakdown—so it’s a good thing you know what to do to fix it. But if changing the fuse doesn’t solve your problem, you may want to head to a garage to get an expert’s help. It may take some time and cost a bit more money, but an expert can make sure all your Mazda’s systems are working properly.