On the CAT side of the DR, you need to use the TRRS pin: The one with four sections? I tried a TRS, and it kept my rig on transmit.
Using TRS cable in serial socket is not a problem in any of the Digirig’s configurations other than one for TX-500.
It sounds like you may be connecting cable to the serial socket on the Digirig side and to the audio (ACC) connector on the transceiver side.
Please provide more details about what revision of the Digirig you use, what is the model of the transceiver and what is the cable.