Quantitative structure-activity relationship (QSAR) approach was carried out for the prediction of inhibitory activity of some novel quinazolinone derivatives on serotonin (5-HT(7)) using modified ant colony (ACO) method and adaptive neuro-fuzzy interference system (ANFIS) combined with shuffling cross-validation technique. A modified ACO algorithm is utilized to select the most important variables in QSAR modeling and then these variables were used as inputs of ANFIS to predict 5-HT(7) receptor binding activities of quinazolinone derivatives. The best descriptors describing the inhibition mechanism are Q(max), Se, Hy, PJI3 and DELS which are among electronic, constitutional, geometric and empirical descriptors. The statistical parameters of R(2) and root mean square error are 0.775 and 0.360, respectively. The ability and robustness of modified ACO-ANFIS model in predicting inhibition behavior of quinazolinone derivatives (pIC(50)) are illustrated by validation techniques of leave-one-out and leave-multiple-out cross-validations and also by Y-randomization technique. Comparison of the modified ACO-ANFIS method with two other methods, that is, stepwise MLR-ANFIS and GA-PLS-ANFIS were also studied and the results indicated that the proposed model in this work is superior over the others.