Knowing solution structures of cyclic peptides is essential for predicting pharmacokinetic properties for drug discovery. Here, we report a dataset and computational workflows to systematically evaluate existing simulation methods that produce cyclic peptide conformers. The dataset was compiled from the literature and contains 68 cyclic peptides and macrocycles with existing solution NMR data. We provide a reproducible and automated computational workflow to quickly compare different cyclic peptide conformer generators with one another and to NMR derived Nuclear Overhauser Effect (NOE) distance constraints. We find that enhanced sampling molecular dynamics methods, such as Gaussian accelerated Molecular Dynamics reproduces experimental evidence well. In contrast, conventional Molecular Dynamics can suffer from a lack of sampling for certain compounds and does not always match with the experimental reference data. Cheminformatics based conformer generators such as OMEGA and RDKit often generate diverse and plausible structures that match the molecular dynamics-based methods, but do not yield relative populations or thermodynamic insights. The presented computational workflow can be easily extended to include new compounds or different simulation methods. This work is developed to be a reference point for future development of improved conformer generators for cyclic peptides to ensure a standardised and systematic evaluation of how well cyclic peptide solution structures are reproduced.