Israeli troops detained 60 people in an overnight raid on a West Bank village as the search continues for the killers of a settler family, Palestinians say.