scala - MQ-like Java library/framework for processing large data with fault-tolerance and re-injecting of large messages -
the requirements are: several services produce lots of requests data streams (1-10 megabytes per request) , several consumers process these requests atomically.
so request data can not split. key requirement if consumer fails process request - request must re-injected queue , sent different consumer. processing of single request may take tens of minutes. consumer may 'die' during processing, sort of task monitoring required.
i considered far
- http://kafka.apache.org/
- http://storm-project.net/
- http://activemq.apache.org/
- http://activemq.apache.org/apollo/
but neither of them seems meet requirements. suggestions?
regarding requirements (> 1 mb per request, processing > 10 minutes) ... still messaging?
i put in database (rdbms). if polling not acceptable, i'd consider triggers/stored procedures in database. stored procedures written in scala work in oracle , postgresql.
Comments
Post a Comment