Data Science

The Difference Between Data Transformation and Data Translation

admin — Sat, 22 Apr 2023 15:12:27 +0000

The Difference Between Data Transformation and Data Translation admin Sat, 04/22/2023 - 22:12

What is Data Translation?

Data translation can be defined as the process of converting volumes of data from one syntax to another and performing value lookups or substitutions from the data during the process. Translation can include data validation as well. One example of data translation is to convert EDI purchase order document data into purchase order database files or even flat files while performing data validation on the source data.

Data Translation is the process of converting data from the form used by one system into the form required by another.

What is Data Transformation?

Data transformation is the process of converting data from one format to another, typically from the format of a source system into the required format of a destination system. Data transformation is a component of most data integration and data management tasks, such as data wrangling and data warehousing.

An example of a data transformation tool is routing a purchase order to a specific process that will perform a data translation to convert the shipping and billing address information to an invoice document. We can also use the translation example from above, transforming an EDI purchase order to a database and then during this transformation, also perform additional processing using formulas such as determining a total number of items or total dollar amount by looping through specific data constructs. Other actions that can be performed during the data transformation process include invoking web services and calling a process. Another transformation example may be to convert character data from one character-encoding scheme to another.

A data transformation tool is not only used for data translation, but a lot more. Data translation is limited to data operations, whereas data transformation combines data operations and process control in a single model.

Via cleo.com

A famous Indian parable "The Blind Men, the Elephant" and the 3 Data Mistakes

admin — Wed, 21 Dec 2022 03:09:06 +0000

A famous Indian parable "The Blind Men, the Elephant" and the 3 Data Mistakes admin Wed, 12/21/2022 - 10:09

A famous Indian parable describes five blind men encountering an elephant for the first time. They each decide to touch the elephant to understand what the animal is like.

“An elephant is smooth and hard, like bone,” says the man who grasped its tusk

“Nonsense, an elephant is soft like leather,” says the man who felt its ear.

“You’re both wrong; an elephant is rough like a tree,” says the man who touched its leg.

The five men begin arguing vehemently, each convinced that he is right. None of them manages to convince the other of anything, except that everyone else is untrustworthy.

This is my favorite fable. It says so much in so few words, about the way we reside in our lonely heads, why alignment is an Everest mountain, how the shape of truth is diamond-faceted.

I see this story often used to describe organization dysfunction — See? This is why Sales and Product keep quarreling!

But hear me out. Recently, I’ve started to see this story as a perfect embodiment of another topic: the 3 most common mistakes teams make in using data.

Ready to explore this elephant with me?

Mistake #1: Rejecting Data that Doesn’t Match Your Beliefs

The obvious thing we — who are not newbies to elephants — can recognize is that none of the blind men are wrong!

An elephant is simultaneously soft like leather, smooth as a bone, and rough like tree bark. It’s also a million other adjectives, because it’s a complex, majestic animal! Such is our world, one where we extol the wise words of Walt: Do I contradict myself? Very well then I contradict myself (I am large, I contain multitudes).

But in a team setting, when you’re locked in a tense battle of wills, it’s hard to remember this. Everything becomes binary.

Consider a data analyst sharing experiment results: Unfortunately, this new redesign decreased user engagement by 5%.

No way, retorts the designer. This redesign is way WAY better. Can’t you see for yourself how much simpler it is? People loved it in the lab. And look at this Twitter feedback! <cue visual of many fire emojis>

I’m embarrassed to admit I have made various forms of the above argument. Like the blind men, I turtled into a narrow definition of the truth: I only embraced data that confirmed what I wanted to believe — This redesign is Awesome!

This is the number one killer of data discipline: Instead of testing my intuition with data, I was seeking data that confirmed my intuition.

In most instances where we bring up conflicting pieces of data, the reality is that both viewpoints are true — there are indeed people who love the new redesign, while on average it causes more people to use the feature less (a lesson I learned the hard way, see Exhibit A here).

Instead of trying to prove one’s opinions, we should aim to broaden our view of reality to incorporate all the data. Only then will we more clearly “see” the whole elephant.

Mistake #2: Selecting Ineffective Methods of Measurement

If none of the blind men were technically wrong about the elephant, then the million dollar question becomes:

What, exactly, is the best way to describe an elephant?

Just now, I asked this question to a pack of kids in my yard. They hollered out answers like Big! Long trunk! Grey! Nobody in this (admittedly limited) study suggested comparisons to bone, trees, or leather.

Perhaps using touch to describe an elephant is unideal. Perhaps limiting said touch to a few inches of surface area on a single occasion is even less ideal.

If 5 people in a research lab swiped around your feature muttering yeah, this is nice, would you conclude that your entire population of 20M users will love it?

When we talk about data, we must accept that every standard of measurement is a proxy for reality. Picking the best proxies to shed light on what you care about is an art, not a science. Every metric you come up with will have shortcomings in conveying the complete truth.

Take Time Spent On App, for instance. This metric is often used as a proxy for answering How worthy of one’s precious time is this?, aka more Time Spent = More Valuable Service. Netflix, TikTok, and Fortnite proudly brandish their time spent numbers.

But take the case of a travel booking site. When I browse for flights or a hotel, I’m typically searching for a good price on a particular date and location. If I spend a lot of time on the site over a few days, what does that reveal? That I’m having fun exploring hotels? That I’m frustrated because it’s taking forever to find options that match my criteria? It’s murky; time spent isn’t super telling in this case.

To know the best way to understand app value or to describe an elephant, we need to know the why. We need a purpose before we need the data.

Help people easily recognize an elephant in the wild if they encounter one is an entirely different purpose than Evaluate whether hunting elephants can be profitable or Determine if wild elephants are thriving.

Once you know your purpose, you must then continually iterate on the best proxies of measurement to give you a picture of reality.

If I’m trying to improve the health of wild elephants, should I take stock of their height or their weight? Their color or their distance travelled? Their lifespan or their herd size? For how many elephants can I get this information easily and accurately? How often can I get updates?

There’s no right answer here; the only way forward is to keep iterating on the best proxies for reality as it relates to our purpose.

Mistake #3: Failing to Turn Disagreements into Learnings

So data is messy and reality skews more complex than we imagine.

What then? Is there anything those blind men could have done to avert the crisis of shattered friendship?

Yes indeed, if only they knew the power of turning disagreements into falsifiable hypotheses, in other words, turning their assumptions into true or false tests.

An example of a good falsifiable hypothesis: Simplifying our onboarding flow will improve new user retention after one week. We can actually make a wager on this! Our whole team can stand around and look at the results of user retention after one week and be exactly on the same page about whether it happened or not.

Contrast this with a less falsifiable hypothesis, like Changing our illustrations will make our product more tasteful. What does tasteful mean? Who gets to judge? We could argue about this all night.

Another way to ask this is: What evidence would convince you that your belief is wrong? Then, listen closely to what the person says and propose a way to collect that evidence. This is what top data-informed companies like Netflix, Google, Amazon do; they love channeling their inner scientist and running A/B tests to test their beliefs.

Even if you can’t do a pure controlled experiment, you can still make concrete guesses about what will happen. Nothing helps a group learn like writing down each person’s bets on a public board and checking the scorecard after the game.

“An elephant is smooth and hard, like bone,” says the man who grasped its tusk.

“Nonsense, an elephant is soft like leather,” says the man who touched its ear.

Imagine if the third man had said, “Let us ask another 100 people what they think, and see what the majority opinion is.”

We’ll all make better decisions — and more friends — this way.

Julie Zhuo (via medium)

The Fallacy of Data-driven Decisions

admin — Thu, 28 Jul 2022 05:11:41 +0000

The Fallacy of Data-driven Decisions admin Thu, 07/28/2022 - 12:11

Entrepreneurs and managers who lead emerging companies often make critical decisions based on imperfect data and a gut feeling.

Most, if not all, companies seek “data-driven” decisions. When it launches a new product, updates its branding, or defines its focus, a company desires excellent market intelligence.

Unfortunately, excellent intelligence rarely exists. Human bias affects data interpretation. But we still have to make decisions. To understand the complexity, consider a couple of decision-making frameworks.

DIKW Flow

Information-action Paradox

In business, it is often the case that the more public a market trend, an opportunity, or a customer segment, the less freedom we have to act.

As the authors of a recent Harvard Business Review article put it, when “data is widely available…others see the same opportunities and risks and respond to them.”

The information-action paradox implies that the freedom to act on information and its public availability are inverse.

A new entrepreneur can make a decision based on relatively sparse public data, an inkling of a trend. She has little information but plenty of freedom to choose.

An incumbent business must often wait for more public data since its managers can rarely act unilaterally but must get buy-in from superiors, peers, and subordinates. These leaders must work through a process of coaxing and convincing, often waiting for additional data. Frequently, such companies have more info for strategic decisions but relatively less freedom to act since competitors will also be aware of the opportunity.

This paradox might lead some managers to act quickly, ahead of competitors.

Diderot Effect

Decisions based on little-known data could lead to opportunity, but they require executives to trust their gut stemming from experiences and feelings.

Human psychology becomes a factor.

Here’s a real-life example. A software-as-a-service company in the process of repositioning itself decided to focus on mid-sized customers. It made the choice with relatively little public information. But because it is smallish, the company had significant freedom to act.

The decision impacted several managers and their departments.

It prompted one manager to conclude she must stop concierge support to 20 of the company’s largest and most profitable customers since those enterprises were no longer the focus.

Ignoring very profitable customers because they do not fit with the company’s perceived strategy is reminiscent of the Diderot Effect in buying.

Denis Diderot was an 18th-century French philosopher. Among his many influential works is a lean essay about a new red robe, titled “Regrets for my Old Dressing Gown, or A warning to those who have more taste than fortune.”

Diderot got a fancy new robe and ultimately redecorated his entire study with expensive art and furniture because the fine robe made the room feel uncoordinated and ugly. Diderot was less happy despite the new possessions.

In the late 1980s, cultural anthropologist and author Grant McCracken used Diderot’s essay to illustrate two human tendencies, which he collectively called “the Diderot Effect.”

Purchase decisions are not always rational; we buy things that match our identity.
A new possession, like Diderot’s robe, can redefine our identity, prompting us to make subsequent and unnecessary purchases that conform to it.

The BBC has an excellent explainer video describing the Diderot Effect.

More recently, author James Clear used the Diderot Effect in his book “Atomic Habits” to describe how personal identity impacts choice and routine.

Thus the Diderot Effect describes buying behavior and habit formation, but it could also affect business decisions.

Hence the SaaS manager was willing to stop servicing 20 of a company’s best customers because that decision matched her perception of the business’s identity.

Balance

The information-action paradox and the Diderot Effect are present when a company makes strategic decisions.

The information-action paradox implies that business leaders should act when they have more freedom, taking advantage of opportunities before their competitors.

But the Diderot Effect implies that we should pause to understand what motivates those decisions.

So here is the good news. Business leaders who recognize these forces can account for them, striking a balance. When considering a strategic decision, look at both: the data and the influences.

Armando Roggio
Source: https://www.practicalecommerce.com/the-fallacy-of-data-driven-decisions

Data Science

The Difference Between Data Transformation and Data Translation

What is Data Translation?

What is Data Transformation?

Tags

A famous Indian parable "The Blind Men, the Elephant" and the 3 Data Mistakes

Tags

The Fallacy of Data-driven Decisions

Information-action Paradox

Diderot Effect

Balance

Tags