CLIPIT-ELASTIC SEARCH CLUSTER

CLIPIT-ELASTIC SEARCH CLUSTER

About this project

Our client accumulated over 2.3 billion news articles and posts from various sources, over the course of more than 15 years. They wanted to expose that rich dataset to analyse data for their clients through an easy to use and familiar web solution, but had no way of accomplishing satisfying performance with the existing technology. They were looking for a big data solution which would provide the needed storage capacity, performance considerations and custom searching language that will offer all the needed options for the end clients.

 

The goal

Realtime searching and analysis on a large dataset

Serving large number of clients simultaneously

Stable and reliable search platform

Provide statistics, reporting, and charts

 

The challenge

Over 2.3 billion news articles and posts from various sources, for a period of more than 15 years with varying size and structure

Achieve satisfying performance results

Possibility to scale for the next several years

The process

Research, comparison of technologies

Selection of best suited technology: Elasticsearch

Design and sizing for an ElasticSearch cluster

Custom solution for parsing and indexing existing archive and live data

Specification

Use cases and scenarios

System architecture

 

Development

REST search API with custom query language

User interface for searching, filtering, aggregation and analysis of result data

Integration with existing portal

 

Testing

Automated unit and integration tests

Deployment & Configuration

Monitoring & Audit logs

The result

Service for searching and analysis of data that is easy to use and integrated in a familiar portal

Standard format and categorization the complete dataset

Enriched dataset with sentiment, language and gender analysis

Distributed and scalable system with high performance

cta: 

HAVE AN IDEA?

HAVE AN IDEA?

Let us know your ideas, we could be able to help.

We are passionate about solving problems, building things and helping the community.

image: 
tag: 
test: 
About us: 
About us

We help our customers go beyond their first intended objectives by building great products to solve their business problems.

hello@iborn.net

+389 2 30 80 261,

+389 2 55 11 437

Varshavska 9A, Str., Postal Address: PO BOX 106, 1020 Skopje, Republic of Macedonia