The following code will reproduce this problem. Just setting the format and setting the editmask.mask is not a problem.
ls_mod = "amt.Format='#,##0.00' "
ls_result = dw_1.modify(ls_mod )
ls_mod = "amt.editmask.Mask='###,###,###,###,###.00'"
ls_result = dw_1.modify(ls_mod )
ls_mod = "amt.editmask.UseFormat='no'"
ls_result = dw_1.modify(ls_mod )
At this point the editmask limits user entry to 9,999.99
The work around is to set the useformat FIRST, and then afterwards set the editmask.mask.
ls_mod = "amt.editmask.Mask='###,###,###,###,###.00'"
ls_result = dw_1.modify(ls_mod )
This only occurs if the editmask.mask was already set in the IDE, and THEN changed as per above.