Archive for February, 2010

Collaboration versus Collectivity

February 9, 2010

Add to FacebookAdd to DiggAdd to Del.icio.usAdd to StumbleuponAdd to RedditAdd to BlinklistAdd to TwitterAdd to TechnoratiAdd to Yahoo BuzzAdd to Newsvine

In previous posts, especially in those related to content mapping, I frequently referred to collective actions and efforts in describing certain concepts, but never elaborated on the exact meaning of these terms. One could think that collectivity and collaboration are identical (they often are mentioned in the same context) as both have something to do with individuals working together. In fact, I find it important to highlight their differences for I expect collectivity to play as vital a role in Web 3.0 as collaboration did in Web 2.0.

Transition

As already understood from popular Web 2.0 applications such as Wikipedia, Google Docs, or WordPress, we define collaboration as sharing workload in a group of individuals who engage in a complex task, working towards a common goal in a managed fashion, and are conscious of the process’ details all the way.

As the number of participants grow however, it becomes apparent that collaboration is not scalable beyond a certain level while remaining faithful to the definition outlined above. Although there is such a thing as large-scale collaboration, what that covers is lots of people having the possibility of contribution but in reality only a few doing so. Mass collaboration goes further by blurring the definition of collaboration so much that it practically becomes just another expression for collectivity.

And when I speak of collectivity, I think of a crowd performing a simple, uncoordinated task where participants don’t have to be aware of their involvement in the process while contributing. The outcome of a collective action is merely a statistical aggregation of individual results.

Different realms

Collaboration and collectivity operate in different realms. Collaboration can be thought of as an incremental process (linear) while collectivity is more similar to voting (parallel). On the figure below, arrows represent the timeline of sub-tasks performed by participants.

Suppose a sub-task like that was the creation or modification of a Wikipedia entry. In this case collaboration proves more effective, as it offers a higher chance of eliminating factual errors during the process, while a collective approach would surely preserve all of them (and offer the one with the fewest). The semantic complexity of a document does not fit the more or less hit-and-miss approach of collectivity.

However, if we decrease the complexity of the content, say, to one sentence, the probability of individual solutions being as ‘good’ as products of collaboration is expected to be equal. Collective approaches therefore suit low-complexity content better.

The synaptic web

What content is of lower complexity than connections within a content network? Different relations such as identity, generalization, abstraction, response or ‘part-of’ require no more than a yes-no answer. Collectivity is cut out exactly for this kind of tasks.

As the creators of the synaptic web concept put it,

With the advent of the real-time web, however, increasingly effective publishing, sharing and engagement tools are making it easier to find connections between nodes in near-real time by observing human gestures at scale, rather than relying on machine classification.

Hence the synaptic web calls for collectivity. What we need now is more applications that make use of it.

Updates

  • Just one day before my post, @wikinihiltres posted an article comparing the efficiency of collective and collaborative approaches to content production through the examples of Wikipedia and Wikinews, concluding that “the balance that ought to be sought is one that continues to accept the powerful aggregative influence, but that greatly promotes collaboration where possible, since collaboration most reliably produces good results”.
Advertisements

Database Options for Content Mapping

February 5, 2010

Add to FacebookAdd to DiggAdd to Del.icio.usAdd to StumbleuponAdd to RedditAdd to BlinklistAdd to TwitterAdd to TechnoratiAdd to Yahoo BuzzAdd to Newsvine

While writing posts on the relation between content mapping and semantic web-related topics, I’m also working out the technical background for a specific content mapping solution.

Organic network of content

Content mapping is a collective effort to organize content into an organic, “living” network. As opposed to technologies that attempt to understand content by semantic analysis, content mapping facilitates understanding by having humans classify the connections in between. It is built on the presumption that conceptualization in the human brain follows the same pattern, where comprehension manifests through the connections between otherwise meaningless words, sounds, and mental images gathered by experience. Content mapping therefore is not restricted to textual content as NLP is. It’s applicable to images, audio, and video as well.

The purpose of content mapping is to guide from one piece of content to its most relevant peers. Searching in a content map equals to looking for the ‘strongest’ paths between a node and its network.

Approach & architecture

The technical essence of content mapping is the way we store and manage connections. From a design perspective, I see three approaches.

  • Graph: Querying neighbors in arbitrary depth while aggregating connection properties along paths. Limited in the sense that it works only on networks with no more than a fixed number of edges on a path, e.g. question-answer pairs.
  • Recursive: Crawling all paths in a node’s network while calculating and sorting weights. Resource hungry due to recursion. Aggregated weights have to be stored until result is returned, and cached until an affected connection is changed.
  • Indexing: Tracking paths as implicit connections on the fly. All implicit connections have to be stored separately to make sure they’re quickly retrievable.

When deciding on an architecture upon which to implement the solution, three choices come to mind.

  • Relational: Traditional RDBMS, mature and familiar. The richness of SQL and data integrity is highly valuable for most web applications, but such advantages often come at the price of costly joins, tedious optimizations and poor scalability.
  • Graph: Fits applications dealing with networks. Despite the structural resemblance with content maps, this genre of databases – being relatively young – lacks certain features necessary for a content mapping solution, such as aggregation along paths.
  • Distributed: Scalability and performance are given highest priority. Consequently, access to resources, and features common in relational databases such as references, joins, or transactions are limited or completely missing.

The following table summarizes the key characteristics of each of the nine approach-architecture combinations.

Graph Recursive Indexing
Relational Costly self-joins in fixed depth Complex, caching required Writing is not scalable
Graph No aggregation along paths Graph architecture not exploitable Implicit connection as separate edge type
Distributed Lacks joins, same as recursive Limited access to resources Needs concurrency management

Finalists

The table above shows that most options have at least one showstopper: either complexity, lack of features and scalability, costly operations or unfitting architecture.

Only two of them seem to satisfy the purpose of content mapping as described in the first section: the graph and distributed implementations of the indexing approach.

  • Even though it’s not the graph approach we’re talking about, this is a combination that exploits the advantages of the graph database to its full extent. By storing implicit connections as separate edges, there’s no need to query paths deeper than one neighbor.
  • In a distributed database there are no constraints or triggers, demanding more attention in regard to concurrency management. Graph structure is not supported on a native level, but scalability and performance make up for it.