SAPO Codebits 2010

Session Title

Distributed programming and data consistency

Abstract

While there is a general perception that all large data processing problems in the world can easily be solved via memcached, a NoSQL database, a Map Reduce solution and thousands of off the shelve computers, there are already many alert signs that it is not so easy (the Twitter whale being just one of them).

There are different data consistency needs and not all space/time data processing graphs are the same.

After understanding how distributed computing is powerful, you may want to know how hard it can be, how to recognize the signs of hardship and have a broad view at the tools available to deal with it.

We will take a look at several data processing problems, separating the easy to partition from the harder ones by pointing to their tell signs. We will then iterate trough solution candidates and respective tradeoffs, from Map Reduce to Locality related optimizations.

If this still sounds to abstract, it means we will also be taking a broad look at (among others) what are and what is the use of:
* Processing Grids;
* Data Grids and "NoSQL" databases;
* MVCC databases, data consistency and compensation mechanisms;
* (The limitations of) Two/Three Phase Commits;
* Coordination, the Paxos Algorithm, Hadoop Zookeeper.

Language

Portuguese

Speakers

Paulo Jorge Bombarda Gaspar

More than 20 years developing software or day dreaming about it. I like web, messaging, storage, fighting for high-performance, optimizing, automating, glueing disparate things together. I also like stepping out of the beaten path whenever it doesn't seem to be the "best path"... whatever "best" means at the time... With so much time around computers, I ended up also having the chance to to most of the common things you can do around computers, like: * assembling them + build and admin computer networks; * hack with many operating systems and computer languages / platforms; * and even the related paperwork, the client facing bits, the occasional customer proposal and negotiations. But I always crave going back to dreaming and building software. I also like Java, SQL, Javascript, C, JSON, Ajax and SOLR, Firebird, JDBM, GWT, NETTY. I even enjoyed doing a bit of x86 assembler and messing with CPU ports, back in the day when we had to, event to render _text_ fast or to sends messages at 9600 bits per second. (Am I turning into a Dino or what?) I like Macs, because being a sysadmin is not that funny to me and because it is the easiest way to have Unix. It seems I like Ruby, not because the Rails hype, not only because of the language itself, but because I end up enjoying a lot of other Ruby made stuff, like SASS, HAML, Compass (the ones I use now) and many other projects and ideas from the Ruby community. Yes, this is a lame "likes bio". I just miss the "rainbows and unicorns" theme to go with it. =;o) An enterprisy CV can be found at: http://pt.linkedin.com/in/paulogaspar7 (more)