So sorry you are experiencing this with the queen. Yes, nothing guarantees that a mom cat will nurse her kittens until they have fully transitioned to eating on their own. I’ve seen a lot of cats put up with them nursing well past the 10 week mark–they are very good and patient mothers and the kittens are way past needing weaned.
You will need to step in and give them proper nourishment, and there’s no use in punishing her into doing it. If she’s out of milk there is nothing you can really do about it. She may have something else going on…or she could be in heat or even pregnant again if she has gotten out or you have a male cat that hasn’t been neutered.
Lucky for you, you have only a few more weeks of bottle feeding…and you can try mixing some wet food in with the milk and see if they like the taste. Make sure you use the pate’ style and it’s very fine and add just a little to the formula. Open up the nipple so it can flow through and this will get them used to the taste. This food should also stick with them longer than just plain formula. Eventually you can transition them to a saucer, but most are too young at 4 weeks to be interested.
If they aren’t taking to it now, then don’t worry about it and try again later. Eventually, you will know when to give them the extra nutrition, because they will act like they are attacking the nipple by grabbing it with their paws and biting and acting like they are starving.