The LA's Finest actress left the talent show after one series, with reports saying she had been dismissed.
She has now told Variety she raised concerns with producers but "felt singled out as being difficult".
NBC and AGT's producers said they took her concerns "extremely seriously", and they "had no bearing" on her exit.
Union was on the panel for the 14th series of the hugely successful show, with an option to stay on for another year.
Producers Syco Entertainment and Fremantle told BBC they and broadcaster NBC enlisted an external investigator to conduct more than 30 interviews to review the issues she raised.
They referred the BBC to media reports saying the investigation concluded that Union's claims were unfounded.
"The investigation has shown that the concerns raised by Ms Union had no bearing on the decision not to exercise the option on her contract," a statement said.
The singer and actress discussed several issues in her interview with Variety, including how Simon Cowell's smoking allegedly made her unwell because she is allergic to cigarette smoke.
The decision about whether to complain was difficult when someone is "coming onto a set and you are literally met with the very definition of a toxic work environment, and it's being carried out by the most powerful person on the production," she said.
She "hesitantly" raised the problem with producers, but was told that "effectively, nothing was going to change", according to Variety.
"I ended up staying sick for two months straight," Union said. "It was a cold that lingered, and turned into bronchitis, because I couldn't shake it. It impacted my voice, which affects my ability to do my job."
Cowell's spokesperson told Variety that "when he was directly informed of the smoking complaint during the first couple of days of the season, he immediately changed his behaviour and the issue was never raised again".
Cowell's spokesperson added: "Simon does appreciate and respect feedback."