Notice, the amount of force applied over a distance. Ow wait a minute! That sound familiar!
W = Fd
The magnet did no work.
The field is there (in a permanent magnet) because of the spin states of the electrons in the magnet.
As the metal approaches the magnet the magnetic field changes.
It changes to a lower energy state.
Energy has been removed from the field.
The field has done work.
The magnet just sat there.
If the magnet was not fixed and it moved to the piece of metal, would you say the metal did work? No. And neither did the magnet. The field did it.
The same is true of gravity. (And I went through this with you a long time ago)
What is potential energy? IS it energy in something? No, it is the energy that would be released by the gravitational field working to bring 2 masses together.
If you fall off a cliff have you done work? No. Has the earth? No. But work has been done. By what? The gravitational field of you and the earth moving to a lower energy state.
In either case work must be done to set things how they were. That work placed the fields in a higher energy state, i.e. it puts energy back into the fields.