SAPO Codebits 2009

Session Title

Distributed programming and data consistency

Abstract

While there is a general perception that all large data processing problems in the world can easily be solved via memcached, a NoSQL database, a Map Reduce solution and thousands of off the shelve computers, there are already many alert signs that it is not so easy (the Twitter whale being just one of them).

There are different data consistency needs and not all space/time data processing graphs are the same.

After understanding how distributed computing is powerful, you may want to know how hard it can be, how to recognize the signs of hardship and have a broad view at the tools available to deal with it.

We will take a look at several data processing problems, separating the easy to partition from the harder ones by pointing to their tell signs. We will then iterate trough solution candidates and respective tradeoffs, from Map Reduce to Locality related optimizations.

If this still sounds to abstract, it means we will also be taking a broad look at (among others) what are and what is the use of:
* Processing Grids;
* Data Grids and "NoSQL" databases;
* MVCC databases, data consistency and compensation mechanisms;
* (The limitations of) Two/Three Phase Commits;
* Coordination, the Paxos Algorithm, Hadoop Zookeeper.

Language

Portuguese

Speakers

Paulo Gaspar

Paulo is a one-job-at-a-time freelance software developer based in Lisbon. Designing and developing software since 1986, he lived and worked abroad for a few years, was an Apache committer (http://commons.apache.org/contributors.html) and performed some international consulting gigs before (finally) becoming a father. With several programming languages, tools and domains under his belt, he spent the last 10 years mostly developing high traffic web applications and large system integration solutions. He is currently obsessed with distributed systems focusing on data consistency and data storage related performance while (and because of) leading the development of a "SOAP-to-Scripting (and back)" integration layer. (more)