The forecast of hurricane trajectories is crucial for population and goods protection. In this work, we propose a fused neural network composed of one neural network using past trajectory data and of one convolutional neural network using reanalysis atmospheric wind fields images. This fused network is trained to estimate the longitude and latitude 6h-forecast of hurricanes and depressions from a large database from both hemispheres (more than 3000 storms since 1979). The average error distance (32.9km) is significantly lower than the baseline (46.5km), and the advantage of the fusion of the two networks is demonstrated.