Jerry, I agree with you. I have only seen it as worn by Royal Ordenance Factory workers.
If you scroll down to pages 9 and 10 in this PDF file you will see a picture of some ROF ladies wearing the patch on their left sleeve of their civilian dress.
http://www.1940.co.uk/archive/Issue82.pdf
I have not yet seen a picture of a member of the HG wearing the patch. Once I do, I would want one of course!
Terry