The Intelligence and Law Enforcement field must deal with a daily flood of data that may or may not be relevant. Simply capturing this enormous data volume and shipping it off to some obscure archive will not work.
To make the problem more difficult, intelligence data almost always comes in various formats (i.e. emails, documents, multimedia and database transactions – or “EDMT”) and may reside across multiple locations, making retrieval and analysis very difficult and slow and making real-time response virtually impossible. In addition to “human-generated” EDMT data, there is streaming EDMT data—time-series, event-oriented data, images and video streams emitted by hundreds or thousands of devices in a network at rapid rates. This sort of data often requires automated, complex correlation prior to human interpretation, and often demands immediate, real-time action. Both EDMT data types have similar characteristics the government point of view:
The military, homeland security, financial, manufacturing, and energy sectors rely on both data types. How data is collected, analyzed, and used is changing rapidly as real-time, sensor-based monitoring applications grow in popularity. In response, the federal intelligence community must prepare for and embrace that change if the government is to remain fully effective in its regulatory, oversight, law enforcement, and public safety roles.
Sensors and monitoring applications—whether placed on a street corner, on a car, inside the electric grid or inside a dialysis machine—produce streaming data that is live, continuously changing, structurally invariant, and voluminous.
Analysts and analytic procedures require speed, efficiency and 100-percent accuracy and transparency to data formats when it comes to the software tools they use to carry out their intelligence work. EDMT® Solution provides breakthrough technology for storing and cross-analysis of all “EDMT” data and is used by the world’s most prominent organizations, some storing and filtering terabytes of “EDMT” data on a daily or hourly basis.
EDMT® Solution manages, analyzes and mobilizes the data irrespective of type and form, unlike most search tools that just search textual data but do not manage the source data and do not provide cross-analysis. EDMT® Solution performs real-time, high-speed ingests of all data types into its own repository, provides high-speed analysis and cross-analysis of all data types using SQL processing combined with Full Text Search and protects data through data retention and auto-deletion procedures.
EDMT® Solution is certified to handle over 1 Petabyte of mixed EDMT data and compressed the data to less than 200 TB. Data Ingest and indexing rate was over 1.5 TB/hour – equivalent to over 1.5 Billion SMS or 150 Million emails per hour or over 10 Billion database transactions per hour. All the data was indexed and ready for search 1-4 seconds after it has been ingested into EDMT® Solution. “Pin-point” queries returned in < 1 sec, and complex queries took 3-300 sec. Visit sybase.com or bmmsoft.com for more details.