Given a smooth oriented manifold $M$ with non-empty boundary, we study the Pontryagin algebra $A=H_\ast(\Omega )$ where $ \Omega $ is the space of loops in $M$ based at a distinguished point of $ \partial M$. Using the ideas of string topology of Chas-Sullivan, we define a linear map $\{\{-,-\}\}: A \otimes A \to A\otimes A$ which is a double bracket in the sense of Van den Bergh satisfying a version of the Jacobi identity. For $\dim(M)\geq 3$, the double bracket $\{\{-,-\}\}$ induces Gerstenhaber brackets in the representation algebras associated with $A$. This extends our previous work on the case $\dim(M)=2$ where $A= H_0(\Omega )$ is the group algebra of the fundamental group $\pi_1(M)$ and the double bracket $\{\{-,-\}\}$ induces the standard Poisson brackets on the moduli spaces of representations of $\pi_1(M)$.