The spatial organisation of chromatin inside the cell nucleus has emerged as a central feature of epigenetic regulation. However, the functional role of this organisation and its relationship with transcriptional regulation are not well understood. Here, we generate chromatin contact maps for zebrafish (Danio rerio) embryos at 2.5 kb resolution and demonstrate that the main three-dimensional (3D) organisational features and the molecular mechanisms behind these are conserved throughout the vertebrate clade. We find that despite the strong conservation of locus-specific organisation at short evolutionary distances, overall, 3D genome conservation between teleosts and mammals is a rare occurrence. However, we find a set of 196 regions of synteny with an exceptional degree of structural conservation of the 3D genome that also feature a high degree of non-coding sequence conservation, suggesting a strong selective pressure to maintain chromatin organisation at these loci. Analysis of duplicated genes retained after the teleost whole genome duplication revealed a preferential unidirectional loss of chromatin organisation associated with the loss of conserved non-coding elements. Finally, in situ hybridisation, bulk and single-cell RNA-seq data demonstrate a functional diversification of gene expression patterns following the rewiring of chromatin with a retention of similar 3D genome structure being associated with higher similarity in developmental expression patterns. Overall, our results strengthen the notion of a functional stratification of the 3D genome in vertebrate genomes and reveal a significant rewiring of chromatin architecture over evolutionary times.