Libyan militias are out of control and secretly holding thousands of detainees, as the government struggles to assert its authority, the UN hears.