Data Strategy

What is Data Gravity?

admin — Sun, 20 Mar 2022 11:52:35 +0000

What is Data Gravity? admin Sun, 03/20/2022 - 18:52

What is Data Gravity?

When working with larger and larger datasets, moving the data around to various applications becomes cumbersome and expensive. This effect is known as data gravity.

The term data gravity was first coined by Dave McCrory, a software engineer, in trying to explain the idea that large masses of data exert a gravitational pull on IT systems. In physics, natural law says that objects with sufficient mass will pull objects with less mass towards them. This principle is why the moon orbits around the earth, and the earth revolves around the sun.

Consider Data as if it were a Planet or other object with sufficient mass. As Data accumulates (builds mass) there is a greater likelihood that additional Services and Applications will be attracted to this data. This is the same effect Gravity has on objects around a planet. As the mass or density increases, so does the strength of gravitational pull. As things get closer to the mass, they accelerate toward the mass at an increasingly faster velocity. Relating this analogy to Data is what is pictured below.

Note: Latency and Throughput apply equally to both Applications and Services

Why does data gravity cause problem?

Data doesn’t literally create a gravitational pull, but smaller applications and other bodies of data seem to gather around large data masses. As data sets and applications associated with these masses continue to grow larger, it becomes increasingly difficult to move. This creates the data gravity problem.

Data gravity hinders an enterprise’s ability to be nimble or innovative whenever it becomes severe enough to lock you into a single cloud provider or an on-premises data center. To overcome the consequences of data gravity, organizations are looking to data services that simultaneously connect to multiple clouds.

How does this all relate back to Database.com? If Salesforce.com can build a new Data Mass that is general purpose, but still close in locality to its other Data Masses and App/Service Properties, it will be able to grow its business and customer base that much more quickly. It also enables VMforce to store data outside of the construct of ForceDB (Salesforce’s core database) enabling knew Adjacent Services with persistence.

The analogy holds with the comparison of your weight being different on one planet vs. another planet to that of services and applications (compute) having different weights depending on Data Gravity and what Data Mass(es) they are associated with.

Data Gravity, Storage, and Cloud Computing

Duplicated data, outside of backups or DR strategies, is wasteful, so maintaining a single big data repository or data lake is the best method to avoid siloed and disparate datasets.

Rather than using a data warehouse, which requires conformity in data, a data lake with appropriate security can handle your raw data and content from multiple data sources.

A data lake with cost-effective scalability seems easy enough, and it can be — depending on the data needs at enterprises. Many organizations have a suitable on-premises data lake, but accessing that data lake from the cloud has several challenges:

Latency – The further you are from your cloud, the more latent your experience will be. For every doubling in round trip time (RTT), per-flow throughput is halved. This can cause a greater likelihood of slowdown, especially for data-intensive data analytics that leverages artificial intelligence and machine learning.
Connectivity – Ordering and managing dedicated network links, such as AWS Direct Connect or Google Cloud dedicated Interconnect, can be costly. Balancing redundancy, performance, and operational costs is difficult.
Support – Operating and maintaining storage systems is generally expensive and complicated enough to require dedicated expert personnel.
Capacity – A location and infrastructure plan and budget for growth are required.

On-premises data lakes can address latency by co-locating closer to public cloud locations and by purchasing direct network connections. Still, the cost is prohibitive for midsized companies who wish to leverage the innovative services of multiple clouds.

Category

IT Consulting

Brief definition of Data Auditing, Data Curation, Data Stewardship and Data Governance

admin — Wed, 09 Mar 2022 14:51:30 +0000

Brief definition of Data Auditing, Data Curation, Data Stewardship and Data Governance admin Wed, 03/09/2022 - 21:51

What is Data Auditing?

Data auditing is the assessment of data for quality throughout its lifecycle to ensure its accuracy and efficacy for specific usage.

Data auditing is the process of conducting a data audit to assess how company's data is fit for given purpose. This involves profiling the data and assessing the impact of poor quality data on the organization's performance and profits

What is Data Curation?

Data curation is the process of creating, organizing and maintaining data sets so they can be accessed and used by people looking for information.

Data curation is an end-to-end process of preparing and managing data so business users can easily understand and readily use it. It is the skill of selecting and bringing together relevant data into structured, searchable data assets that are ready for analysis.

The ultimate goal of data curation is to reduce the time from data to insights. With the growing amount of data in organizations today, data curation is becoming essential. Without it, business users can neither locate useful data nor use it to its maximum potential.

What is Data Governance?

Data governance (DG) is the process of managing the availability, usability, integrity and security of the data in enterprise systems, based on internal data standards and policies that also control data usage. Effective data governance ensures that data is consistent and trustworthy and doesn't get misused.

What is Data Stewardship?

A data steward is an oversight or data governance role within an organization, and is responsible for ensuring the quality and fitness for purpose of the organization's data assets, including the metadata for those data assets.

Data stewardship is not necessarily an information technology function, nor should it necessarily be considered to be a full-time position, although its proper execution deserves an appropriate reward. Data stewardship is a role that has a set of responsibilities along with accountability to the line-of-business management.

Category

Cloud – big data

What is a Data Pipeline?

admin — Mon, 20 Dec 2021 16:23:48 +0000

What is a Data Pipeline? admin Mon, 12/20/2021 - 23:23

Data Pipeline Definition

A data pipeline is the series of automated, consecutive data processing steps involved in ingesting and moving raw data from disparate sources to a destination. Data pipeline software facilitates the seamless, automated flow of data from one system to another, with common steps including: aggregating, augmentation, data transformation, enrichment, filtering, grouping, and running algorithms. A common data pipeline example is the etl data pipeline.

What is a Data Pipeline?

Data pipeline is a broad term referring to the chain of processes involved in the movement of data from one or more systems to the next. Data pipeline tools and software enable the smooth, efficient flow of data; automate processes such as loading data, processing data, extraction, transformation, validation, and combining; protect data integrity; and prevent bottlenecks and latency.

The steps that occur in between the data sources and the destination depend entirely on the use case. A simple data pipeline may only include a single, static data source, extraction, loading, and a single data warehouse.

A complex data ingestion pipeline may involve ingesting and processing multiple data streams in parallel, real-time data sources, transformation, training datasets for machine learning, a visual analytics destination, and multiple pipelines that in turn feed into other pipelines or applications. The best data pipeline solution depends on the nature of the project and business objectives.

Data Pipeline Solutions

A well managed data pipeline infrastructure is a crucial element in data science and data analytics. Data flow is susceptible to disruption, and useful analysis is dependent upon data reaching its intended destination uncorrupted and in a timely manner. An optimized data processing pipeline facilitates reliable delivery of data sets that are centralized, structured, and accessible to data scientists for further analysis.

Some of the most popular data pipeline architectures include:

Batch Processing: Batch data pipeline processing is ideal for work that does not require real-time data. Large amounts of data are moved at consistent intervals.
Real-Time: Also known as stream processing, a real-time pipeline is ideal for streaming data that is being created in real-time.
Cloud Native: A cloud native pipeline is hosted in the cloud and works with cloud-based data, relaying on the hosting vendor’s infrastructure. This is particularly useful for time-sensitive business intelligence applications.
Open Source: Open-source is best for teams that are looking to lower upfront costs and also have data engineers with the technological expertise to develop and modify the public tools available in an open-source pipeline.

Building a Data Pipeline

Some companies have thousands of different data analysis pipelines running concurrently at any given moment. But the most basic steps for building data pipelines typically include, but are not always limited to: identifying data sources; extraction and joining data from disparate sources; data categorization; standardization; data cleansing and filtering; loading data into the destination; and automating the process so that it runs continuously and on schedule.

Data pipeline monitoring tools should be integrated into the architecture to preserve data integrity and alert administrators of failures such as network congestion or an offline destination.

What is a Big Data Pipeline?

Data pipelines in big data are pipelines developed to accommodate the volume, velocity, and variety of big data. Big data analysis pipelines often have a stream process architecture that is scalable, can capture and process data in real-time, and can recognize both structured and unstructured data formats.

An example of big data pipelines is interactions on social media. A single post on a social media platform could generate a series of pipelines branching off into multiple other pipelines, such as a sentiment analysis application, a word map chart application, and a social media mentions counting report.

Source: omnisci

Category

Cloud – big data

DATA SILOS: THE ACHILLES HEEL OF LARGE ORGANIZATIONS

admin — Thu, 16 Sep 2021 17:55:15 +0000

DATA SILOS: THE ACHILLES HEEL OF LARGE ORGANIZATIONS admin Fri, 09/17/2021 - 00:55

DATA SILOS: THE ACHILLES HEEL OF LARGE ORGANIZATIONS

Quick question: how do you determine strategic choices for your business?

While qualitative decision making is a viable approach, you also want to have an understanding of the underlying numbers to help you make the most informed and optimal choice.

In the modern age, companies are unable to come up with decisions without relying on accurately collected data. The biggest obstacle to using precise data analysis is not even the technology or skills required. It is accessibility or rather inaccessibility to the data. In fact, 47% say data siloing and accessibility are the biggest challenges to gaining insight from marketing data.

May companies still suffer from data silos, which are a big blocker for good decision making. Read on to learn how to break down data silos in your company.

What Exactly Are Data Silos?

A data silo, also known as an information silo, is a repository of information in an organization’s department that remains under the control of the said department but is isolated from the rest of the organization.

Much like grain in a farm is sheltered from the elements, a business silo is a mentality or culture that prevents team members from one department from sharing data and information, developments or even technologies with members from other departments.

Are data silos a natural way of doing business?

While data silos seem like a normal business practice, they harm your company’s growth. They prevent you from reaching your full potential since data is locked away from general use or generalized access, thus impeding productivity.

Going back to the physical silo analogy, locking up grain in a silo means it is unusable for its intended purpose unless it is moved from the silo into the world. The solution to the big silo problem is finding a way to move the data from isolation. This ensures data integration among all departments in your organization.

Why Do Data Silos Occur?

Isolated islands of information result from a wide variety of factors. Here are the most common scenarios from organizations suffering from data silos:

1. Technological

Without access to proper technology, data cannot pass smoothly between several departments in a company. Specialized applications for fast and convenient data sharing are required. Even so, members from all departments need to be well trained in using the technology.

2. Company Growth

At times, a company may grow to a point that sharing information becomes difficult. It may be due to many departments, offices around the globe, or just too many employees. A company growing too fast may also result in structural issues.

3. Workplace Culture

Unhealthy competition or animosity between teams may lead to those departments withholding information from one another instead of working in unison.

4. Vendor Lock-In

Vendor lock-in is where a vendor wants to keep you within their cloud platform. Companies should therefore utilize multi-cloud data management instead of relying on only one vendor.

5. Political Reasons

Why do we keep secrets from even some of our friends? It is because the leak of information could result in others misusing that information, even accidentally. The same happens in organizations, but instead of secrets, teams keep information from one another.

The Consequences of Data Silos

While they seem harmless on the surface, data silos are like a hidden iceberg waiting to sink your company.

Data silos exist in virtually all companies. They tend to occur, especially in large organizations with several departments that have different priorities, goals, and responsibilities. A common factor lies within the very core of our organizations, the company’s structure.

Several departments within an organization are necessary for heightened focus and effectiveness. However, if the boundary between departments gets too thick, the free flow of information is soon impeded. It may result in:

1. Slowing down your business

We live in a fast-paced environment where we have to make informed decisions quickly. Waiting for information to be gathered from all departments then processed will have you lagging behind your competition.

2. Negatively impacting crucial areas of your business

Inaccurate spreadsheets hamper your financial planning and budgeting. Poorly organized data will overburden your IT staff. Incomplete performance management data from your HR guys will make it hard to recognize and reward the top employees or even notice if there’s poor team culture. You can remedy this by actively tracking contributions in real time using the right performance management software.

3. Limited communication and collaboration

The existence of data silos within a company means that your employees have limited access to information. Therefore, they cannot utilize the full value of the collected data. They also miss out on the chance to work together with a similar goal in mind.

4. A decrease in the quality and credibility of data

Of course, one of the biggest setbacks of data silos is this. Isolated data is useless, as it quickly becomes outdated or inaccurate. Such data can majorly affect the sampling or analysis of your company’s progress, rendering the findings inaccurate.

5. Storage space and accuracy

In the absence of a central system of data storage, employees who need quick access to data will save copies for themselves. Ten guys storing copies of the same information eats up your storage budget while still making it hard to figure out whose version is the latest.

Five Ways to Break Down Data Silos

To get the most out of your data, you need to access and analyze it in a fluid manner. That means getting rid of the data silos that, to be honest, most of the people working in your organization are not even aware of.

Now, how do you do this?

After that much pessimistic talk about data silos, you will be pleased to know that knocking down data silos is not as hard as many business owners think.

Here are several strategies that organizations use to eliminate data silos:

1. Encourage communication and collaboration

Most, if not all data silos, are created due to a lack of communication between various departments in an organization. For example, your marketing guys may be storing data on Google Drive while their colleagues in the sales department might be using Dropbox. This way, both teams may be unaware of where the other is storing their files. All this is while they are supposed to be working together in the first place.

You can get rid of such isolated data by encouraging your employees to collaborate in cross-functional teams. Using a good project management software is key to ensuring this.

2. Unify technology platforms

Nowadays, you would be surprised at just how many technologies are in use within a single department, let alone a whole organization. A department may have more than five databases in use, with many applications spread across them. It leads to an endless tangle of data silos due to incompatible technology stacks that can’t be integrated.

Simplifying and consolidating your data management systems to a unified central system will work wonders in eliminating data silos. Start by consulting your employees on their data collection methods. Afterward, decide on which systems should be merged or eliminated for a more efficient solution.

A unified view of your data also allows you to conduct more thorough customer analysis, this is especially true if you are running an online business such as an SaaS business where quality and quantity of customer data are paramount to success and strategic decision making.

3. Sift through outdated data

Yes, to create a data management solution that is usable, you have to go through months of outdated, isolated data. It is to ensure that the data you have is updated and accurate.

An easy way to do this would be having your entire company work together through it. While you will be able to weed out any unusable data, it doubles up as a team-building activity. You can then ensure your employees understand how to handle data and the data storage platforms which are approved.

For example, having all your sales data centralised and kept up to date makes it easier for sales managers to monitor each sales rep activity.. You can even use these sales KPIs as a starting point.

4. Move data into a central repository

How about moving all the data to a repository such as a data warehouse or a data lake? It is an ingenious way to exterminate data silos since all your employees will be able to access the data whenever they need it.

The extract, transform, and load process is the most common method of compiling multiple sources of data into a single target database. You can learn more about the ETL process here.

5. Avoid vendor lock-in

Vendor lock-in is a serious problem facing organizations that store data in the public cloud. Vendor lock-in makes migrating data from one platform to another both expensive and time-consuming. Vendors can also use golden handcuffs to incentivize you not to leave their program at all.

To avoid such data silos, you need to use a flexible solution that gives you data autonomy or the ability to transfer your data out of the cloud whenever you want. You can even use a hybrid cloud strategy to remedy this.

Conclusion

As organizations grow, data silos increasingly become a significant threat. Many companies cannot reach their full potential because of restricted access to their own data. The good news is that there are many techniques you can use to get rid of this real headache.

Preventing data silos will allow your organization to share data among teams to help drive sales. When you take steps to deconstruct your data silos, you will witness results almost immediately.

Via clicdata

Category

Cloud – big data

Data Strategy

What is Data Gravity?

What is Data Gravity?

Why does data gravity cause problem?

Data Gravity, Storage, and Cloud Computing

Tags

Brief definition of Data Auditing, Data Curation, Data Stewardship and Data Governance

What is Data Auditing?

What is Data Curation?

What is Data Governance?

What is Data Stewardship?

Tags

What is a Data Pipeline?

Data Pipeline Definition

What is a Data Pipeline?

Data Pipeline Solutions

Building a Data Pipeline

What is a Big Data Pipeline?

Tags

DATA SILOS: THE ACHILLES HEEL OF LARGE ORGANIZATIONS

What Exactly Are Data Silos?

Are data silos a natural way of doing business?

Why Do Data Silos Occur?

1. Technological

2. Company Growth

3. Workplace Culture

4. Vendor Lock-In

5. Political Reasons

The Consequences of Data Silos

1. Slowing down your business

2. Negatively impacting crucial areas of your business

3. Limited communication and collaboration

4. A decrease in the quality and credibility of data

5. Storage space and accuracy

Five Ways to Break Down Data Silos

1. Encourage communication and collaboration

2. Unify technology platforms

3. Sift through outdated data

4. Move data into a central repository

5. Avoid vendor lock-in

Conclusion

Tags