If you consummated the marriage, i.e. had sexual intercourse with the woman, then it is prohibited to marry her permanently. But, if the temporary marriage contract with the first man was invalid for some reason, e.g. if it was without mentioning the dower, then you can marry her again.