Learning robot behaviors in simulation has the advantages of being faster, safer and cheaper than learning on a real system. In exchange, the problem of Simulation Optimization Bias is introduced, which is caused by discrepancies between the model and the real system, commonly referred to as the Simulation-to-Reality gap. More accurate models can be designed by experts for a specific use-case. Still, this endeavour is time-consuming, expensive and thus makes robot learning less accessible to a broader audience. Furthermore, it trades generalizability for accuracy. State-of-the-art approaches for sim2real robot learning are often based on "general-purpose" simulations and domain randomization methods using probabilistic models to increase the robustness of the simulation and to guide the exploration of the real system. The iterative, bi-level nature of the sim2real framework encourages a dynamic multi-modal objective landscape of the simulation optimization problem that changes in every iteration of the sim2real loop. Due to their remarkable performance on continuous multi-modal benchmark problems, we argue that using a Multi-Modal Evolutionary Algorithm is capable of finding and maintaining a diverse set of models throughout sim2real iterations. We hypothesize that explicitly encouraging model diversity could lead towards learning more versatile and robust robot behaviors in simulation. To verify the approach, we will test the algorithm in a simple two-dimensional obstacle simulation and compare the performance with state-of-the-art sim2real approaches. For the final evaluation, we aim to perform comparative simulation-to-simulation and simulation-to-reality experiments on a 6-DoF robot arm with a two-finger gripper in a robotic manipulation scenario.