Verstappen ultimately finished sixth, one place ahead of Hamilton, in a race determined by tyre management as the top 10 on the grid remained in the same order.
With both drivers attempting to eke out the medium tyres they had switched to under a red flag on the opening lap, Mercedes made the first move in bringing seven-time world champion Hamilton into the pits on lap 51.
But the Briton was not issued with an 'out-lap critical' message and rejoined behind Verstappen when Red Bull pitted him one lap later, prompting a frustrated Hamilton to query: "Why didn't you tell me out-lap was critical?"
Asked by sister title GP Racing after the race to explain what happened, Wolff replied: "That was a miscommunication first between us on the pitwall that we got that wrong.
"It should have been an 'out-lap critical', trying to undercut.
"But then there was a debate whether any out-lap would be enough from the new tyre. And so the message he got was at best confusing but probably wrong.
"It should have been an out-lap critical and the worry in the background was that if we thrashed that tyre in a single lap, then what would happen later.
"But in summary wrong message to Lewis, this was the team's fault."
Asked to explain why Hamilton had waited so long in the race before opting to pit, as he had long since opened up a gap to the pursuing RB of Yuki Tsunoda, Wolff later added: "We wanted to be close to Verstappen and then do the undercut, which obviously we got the messaging completely wrong."
George Russell elected not to pit and continued to the flag on the medium tyres he had taken under the red flag, triggered by Sergio Perez clashing with Kevin Magnussen, with the Briton ultimately surviving pressure from Verstappen to finish fifth.
The leading quartet of Charles Leclerc, Oscar Piastri, Carlos Sainz and Lando Norris all started on the medium tyre and took the more durable hard tyres under the red flag which could more comfortably run to the end.
Wolff admitted that "the moment the crash happened, it was clear that we were really on the back foot" and believes there was little Mercedes could have done to alter the trajectory of its race.
"Fundamentally, I don't think there is a lot you can change in Monaco," he said.
"You pretty much end where you start."
