UN human rights chief says forces sent to quash violence in Rakhine state were reported to be targeting Muslims.