Artificial intelligence (AI) mistakes are creating challenges for Australian businesses, with almost half of workers reporting that the most significant AI error they encountered had consequences extending past a minor inconvenience, including financial, compliance or reputational impacts.
New research commissioned by Appian (Nasdaq: APPN) found that 60% of Australian workers had personally identified an AI error in the workplace, raising questions about how effectively organisations are managing the risks associated with the technology.
The study found that one in five workers said the most serious AI mistake they encountered resulted in a customer service issue, while 13% reported operational disruption.
Despite the potential consequences, more than one in five respondents said employees in their organisation are discouraged from questioning AI recommendations or decisions. Just 14% said they fully trust the AI-generated outputs used at work.
Charlie Hutchinson, SVP Asia-Pacific and Japan at Appian, said organisations cannot expect workers to embrace AI if they do not trust its outputs or feel able to question them.
“It is hardly surprising that complete trust in AI is so low when workers are catching it making mistakes and are discouraged from questioning its outputs,” said Hutchinson.
“For businesses, mistrust creates a lose-lose situation. Employees may avoid AI or spend time checking everything it produces, wiping out the productivity gains it was meant to deliver. Alternatively, they may continue using an output they suspect is wrong because they do not feel able to challenge it. AI needs to operate within the same governance processes and controls as the rest of the business, with people able to step in before a significant error occurs.”
Workers concerned about accuracy and accountability
The possibility of AI making incorrect decisions was the leading concern for a quarter (25%) of Australian workers, followed by data privacy and security risks (22%) and a lack of transparency around how AI reaches decisions (20%).
These concerns are influencing attitudes towards the use of AI in more consequential business processes, particularly those involving customers and employees.
An overwhelming 94% of workers opposed allowing AI to make decisions affecting people without restrictions or human oversight.
The research found that senior management also remained more trusted than AI, with 52% of respondents expressing greater confidence in senior leaders compared with 29% who placed more trust in AI.
“Every visible error gives workers another reason to question whether AI should be trusted with more consequential work,” said Hutchinson. “Employees are often the last line of defence before an incorrect output reaches a customer or becomes part of a business decision. If they are discouraged from speaking up, organisations remove one of their most important safeguards.”
AI adoption is outpacing governance
Separate global research from Harvard Business Review Analytic Services, sponsored by Appian, suggests businesses recognise the importance of managing AI risks but have been slower to implement the necessary controls.
While 92% of respondents agreed that AI agents require rules-based guardrails to operate safely and effectively, only 48% said their organisation had established them.
The findings highlight the importance of governance as organisations expand AI use into operational workflows, where incorrect outputs can influence decisions and create additional business costs.
“Businesses know AI needs rules, but many are deploying it before those rules are in place. Doing so without the right controls can quickly lead to unexpected token costs, operational disruption and accountability gaps,” said Hutchinson. “Governance cannot be bolted on after AI is already influencing decisions. AI needs to be embedded within governed business processes that control what it can access, define the rules it must follow, and determine when a decision must be handed to a person. That gives employees the visibility and confidence to use AI while allowing businesses to capture its productivity benefits without giving up accountability.”
Employees want greater control over AI decisions
Australian workers expressed strong support for maintaining human involvement in decisions with direct consequences for employees or customers.
Two-thirds (66%) believed these decisions should either be reviewed and approved by a person or made entirely by people. Another 28% supported limiting AI to low-risk decisions.
The findings suggest that organisations seeking to expand their use of AI will need to address employee concerns about reliability, oversight and accountability.
“Australian workers are rejecting unchecked AI. Businesses will only realise the full value of AI if employees build trust, are willing to use it within critical processes, and confident enough to act on its outputs. That requires governed processes where AI recommendations and autonomous agent outputs can be systematically reviewed, validated, or overridden and human judgement remains part of important decisions,” concluded Hutchinson.
For more information about using AI automation within governed business processes, visit Appian.
