This work explores weakly supervised training of discrimi-native linear classifiers. Such features-rich classifiers have been widely adopted by the Natural Language processing (NLP) community because of their powerful modeling capacity and their support for correlated features , which allow separating the expert task of designing features from the core learning method. However, unsupervised training of discrim-inative models is more challenging than with generative models. We adapt a recently proposed approximation of the classifier risk and derive a closed-form solution that greatly speeds-up its convergence time. This method is appealing because it provably converges towards the minimum risk without any labeled corpus, thanks to only two reasonable assumptions about the rank of class marginal and Gaussianity of class-conditional linear scores. We also show that the method is a viable , interesting alternative to achieve weakly supervised training of linear classifiers in two NLP tasks: predicate and entity recognition.