We report two new cyanido-bridged Fe(II)-Ag(I) coordination polymers using different acetylpyridine isomers, {Fe(4acpy)$_2$[Ag(CN)$_2$]$_2$} 1 and {Fe(3acpy)[Ag(CN)$_2$]$_2$} 2 (4acpy = 4-acetylpyridine; 3acpy = 3-acetylpyridine) displaying thermally and photoinduced spin crossover (SCO). In both cases, the acetylpyridine ligand directs the coordination polymer structure and the SCO of the materials. Using 4-acetylpyridine, a two-dimensional (2D) structure is observed in 1 made of layers stacked on each other by silver-ketone interactions leading to a complete SCO and reversible thermally and photoswitching of the magnetic and optical properties. Changing the acetyl group to a 3-position, a completely different structure is obtained for 2. The unexpected coordination of the carbonyl group to the Fe(II) centers induces a three-dimensional (3D) structure, leading to statistical disorder around the Fe(II) with three different coordination spheres, [N$_6$], [N$_4$O$_2$], and [N$_5$O]. This disorder gives rise to an incomplete thermally induced SCO with a poor photoswitchability. These results demonstrate that the choice of the acetyl position on the pyridine dictates the structural characteristics of the compounds with a direct impact on the SCO behavior. Remarkably, this work opens interesting perspectives for the future design of Fe-Ag cyanido coordination polymers with judiciously substituted pyridine ligands to tune the thermally and photoinduced SCO properties.