I'm getting a new computer to learn CUDA programming.

OK, the release of the GTX960 video for about $200 got me thinking.  I've been waiting for sufficient computing power to fall into my price range so I can get a good feel for the state of Artificial Neural Networks.  ANNs have a long history dating back to the perceptron.  From that early start in the late '50s things went sour and symbolic reasoning took center stage for Artificial Intelligence.  Symbolic reasoning itself has had problems as chronicled in the Wikipedia page AI Winter.  Now it looks like Connectionism is back and GPUs are helping in the process.

The human brain has about 100,000,000,000 neurons.  Each neuron typically connects with 1,000 to 10,000 other neurons or other cell types.  Only governments can afford that kind of computer power right now.  It certainly takes more than the 10 watts our human brain takes.  What's a mere hobbyist to do?

Here's the thing, many specialized applications based upon the connectionist model are coming in the reach of hobbyist.  The Tesla K10 has 2 1536 core GPUs and uses a PCI Express 3 x16 connection.  It was released in 2012.  You can still buy one but it will set you back $2,000.  I've ended up buying a GTX970 in a new custom computer and this end of life cycle machine cost me about the same.  Heck, If I can trust Game Debate the GTX970 I bought as a 51% improvement over the Tesla K20, at least for games.

The current hot ANNs are called Deep Convolutional Neural Networks.  ImageCast is a very good example of 2012 cutting edge.  The paper says: "Our network takes between five and six days to train on two GTX 580 3GB GPUs. All of our experiments suggest that our results can be improved simply by waiting for faster GPUs and bigger datasets to become available." and I believe they were right.  I'll find out soon.

I'm a bit concerned about the various PCI Express implementations.  If I'm understanding what's going on correctly neither the motherboard, the processor, nor the GPU are using the PCI Express 3 connectors efficiently.  I say this because many YouTube videos compare the x4, x8, and x16 slot performance, showing the difference isn't all that much.  The Z97 motherboards can only handle one card at x16 and if you put in a second card both get downgraded to x8.  The X99 motherboards can handle two cards at x16 and one at x8 or four at x8 provided you get a 40 lane CPU .  The X99 motherboard, processor, and DDR4 memory is just too expensive right now and even if it wasn't as long as the video cards aren't using the x16 efficiently it doesn't really matter. 

I may drop in a second GPU depending upon video performance while training a network but I'll likely wait awhile.  Another couple of years and the X99 will be cheap.  By that time I'll know if cuDNNs are working well enough I won't get frustrated.