You are on 12.5.4, so Direct I/O is not available. However, you may want to confirm that the O/S is not attempting to cache the device, when dsync is turned on or off. As you are on 32-bit Windows, the device might be too large to cache in a 32-bit environment And this could be contributing to the problem.
We noticed in 15.7 at least, with Direct I/O off on temp devices, NTFS tried to cache the temp devices (actually ANY device with Direct I/O off). Under ECC these were very large and started to push the ASE out of memory. Windows obviously recognizes MSSQL devices and does not cache them. But with ASE, you must be explicit. The workaround was to turn Direct I/O on, even for temp devices, to prevent caching by the O/S.
Although you you are still on 12.5.4, I wonder if the 32-bit O/S is trying to cache the device. You may want to try running the O/S with /3GT flag (if my memory is correct) to give more data space to processes (gives only 1GB to the O/S instead of the default 2+2 split - assuming you have 4GB of memory on the box) and check if setting Dsync on or off helps.
The other solution is to get to 64-bit, supported ASE and O/S ASAP.
Chris