The team at FusionStorm NYC has been leveraging automated data collection for years.  Over the past two months we have decided to take it to the next level by developing an architecture that would allow us, our partners and our clients to easily mine the information collected.  We have broken the project into simple phases. 

Phase 1:  Develop a data repository (StormDB) and web based architecture that will allow us to easily aggregate information collected from a number of distinct tools and correlate the information thus providing greater value.  This phase is nearing an Alpha release.

Phase 2:  Begin to normalize and aggregate information across the repository.  Our goal here is to help our customers understand real world industry averages by mining our repository.

Phase 3:  Distributed data collection process.  In phase 1 & 2 data is collected using automated collection tools, that data is then packaged in a variety of formats (e.g. - csv) and shipped to the NYCStorm server to be parsed and inserted into the StormDB.  Our goal in phase 3 is to parse and aggregate the data at the edge location leveraging an agent architecture.  Our goal here is to reduce the amount of data shipped to the NYCStorm server and to enhance the aggregation process.  We also plan to provide a local presentation layer in phase three.