We address a variant of Zariski Cancellation Problem, asking whether two varieties which become isomorphic after taking their product with an algebraic torus are isomorphic themselves. Such cancellation property is easily checked for curves, is known to hold for smooth varieties of log-general type by virtue of a result of Iitaka-Fujita and more generally for non $\mathbb{A}^1_*$-uniruled varieties. We show in contrast that for smooth affine factorial $\mathbb{A}^1_*$-ruled varieties, cancellation fails in any dimension bigger than or equal to two.