Bin picking is a widely spread application in industries and its automation through robots generally requires object instance-level segmentation and 6D pose estimation. State-of-the-art computer vision algorithms for these tasks are deep learning-based and require large datasets of diversified annotated images at the instance level, which are prohibitively expensive to acquire. In this paper, we make use of PickSim, a newly developed Gazebo-based dynamically configurable open-source pipeline, and introduce a dataset of simulated data, namely FruitBin, for the challenging task of fruit bin picking. FruitBin contains more than 1M images and 40M instance-level 6D pose annotations over both symmetric and asymmetric fruits with or without texture. Rich annotations and metadata (including 6D pose, segmentation mask, point cloud, 2D and 3D bounding boxes, occlusion rate) allow the tuning of the proposed dataset for benchmarking the robustness of object instance segmentation and 6D pose estimation models (with respect to variations in terms of lighting, texture, occlusion, camera pose and scenes). We further propose three scenarios presenting significant challenges of 6D pose estimation models: new scene generalization; new camera viewpoint generalization; and occlusion robustness. We show the results of these three scenarios for two 6D pose estimation baselines making use of RGB or RGBD images. To the best of our knowledge, FruitBin is the first dataset for the challenging task of fruit bin picking and the biggest large-scale dataset for 6D pose estimation with the most comprehensive challenges, tunable over scenes, camera poses and occlusions.