I suggest doing some research/talking to your doctor, because you need to determine if your baby is allergic to:
lactose (milk sugar)
casein (milk protein)

then talk to me. my mother is an expert in these things.