Lots of data; 7 tables. Enjoy!

Subscribe now

background:

I host my portfolio on Substack, a platform that acts as a marketplace for electronic newsletters. It’s a space where real money can be made for those willing to put in the effort, with exposure being the key to success. So there are clear incentives for some users to cheat the system.

This idea struck me as I browsed Substack, curious about how they decided which publications to show/offer me. It became apparent that three factors played a role: keywords and categorical relevance, publication age, and subscriber count.

I confess—I have a bit of a criminal mind. This sparked the idea for a realistic analysis case study for data analysts working at a similar marketplace. Rest assured, all the data here is fictional and purely for practice—no accusations, no harm, just fun and an opportunity to learn.

Share novsto

2 options:

In here, you’ll get the chance to analyze data. You can either:

1. Dive right in with 3 focused tasks to solve the case directly.

2. Opt for a more gradual approach, with thought-provoking questions to build your business, product, and critical thinking skills, along with analysis and visualization tasks to explore and practice further.

The choice is yours.

Share

the data

• Table Name: USERS
  General idea: table with all users and when they created their user.
  Granularity: one record per userID.

• Table Name: USERPUBLICATIONS
  General idea: table connecting between users and their publications.
  Granularity: each user may have more than one publication.

• Table Name: PUBLICATIONS
  General idea: table with all publications, when they were created, deletion datetime if applicable.
  Granularity: one record per publicationID.

• Table Name: PUBLICATION_CATEGORIES
  General idea: label category assigned to describe each publication.
  Granularity: one record per publicationID.

• Table Name: POSTS
  General idea: one record of postID per publicationID, alongside other data related to the post.
  Granularity: one record for every postID, per publicationID.

• Table Name: EMAILED_POSTS_STATS
  General idea: Information concerning the emailed post which was published and then sent to subscribers.
  Granularity: one record per postID as each postID can be sent/emailed only once.

• Table Name: SUBSCRIBERS
  General idea: dimensional table which records per combination of subscriber and publicationID, details about the subscription.
  Granularity: one record per registration of an email to a / per publicationID, noting time and type of subscriber.

• The entire folder. All data/documents available in that folder.

CREATE DATABASE eletters;
USE DATABASE eletters;

1. create USERS table

2. USERS table data (part 1 of 2)

3. USERS table data (part 2 of 2)

4. create USERPUBLICATIONS table & load bulk

5. USERPUBLICATIONS table data csv

6. create PUBLICATIONS table & bulk load

7. PUBLICATIONS data table csv

8. create PUBLICATION_CATEGORIES table & bulk load

9. PUBLICATION_CATEGORIES data table csv

10. create POSTS table

11. POSTS table data

12. create EMAILED_POSTS_STATS table & bulk load

13. EMAILED_POSTS_STATS data table csv

14. create SUBSCRIBERS table & bulk load

15. SUBSCRIBERS data table csv

Subscribe now

THE SCENARIO

Option #1: dive right in

Imagine it’s the start of your weekly planning session with your manager. You are given the following tasks to tackle:

assist the Head of Marketing & the Product team

The Head of Marketing wants to launch a blog series about creating successful publications. This is in response to user requests. They want to appear on the company’s website/application among the top-20 recommended publications (details of how the company ranks them is described below).

Thus, the first focus will be on subscriber growth:

• Identify what characterizes some publications which attracted and grew their subscriber base more successfully compared to their peers.

• Provide insights & examples that the Head of Marketing and the company’s Content Writer can share.

Not unrelated, you are requested, in parallel, to provide improvements to the ranking logic for the top-20 recommended publications. The Product team awaits your ideas.

• Review the current methodology for determining the top-20 publications.

• Suggest modifications to the ranking system that will better align with the company’s goals.

• Briefly explain the rationale behind your recommendations.

the company’s current ranking methodology for top-20 publications

Here’s how the company currently determines the Top-20 Publications which it displays on its homepage both globally and per category:

1. Eligibility Criteria:

   • Publications must not be deleted.

   • Publications must be at least over 90 days old.

   • Publications must have at least one new post in the last 120 days.

2. Ranking Logic:

   • Publications are ranked primarily by their total number of subscribers as of the current date (feel free to assume the current date is 2021-07-01).

   • In case of a tie, the earliest-created user (publication-owner) is ranked higher.

the CEO’s concerns for marketplace-integrity

The CEO is concerned about fraudulent activity within the marketplace, which damages its reputation and revenue. Your task is to:

• Spot the cheater(s). Look for patterns that suggest abuse or cheating.1

• Briefly explain what/how you identified and why it constitutes cheating.

Share

ADDITIONAL PRACTICE: ANALYSIS, VISUALIZATIONS, BUSINESS + PRODUCT + CRITICAL THINKING

Option #2: Warm Up / Additional Practice

Lots of data, lots of tables, lots of fun. You have here all you need for plenty of practice.

The key to make the most out of it is not by just technically practicing SQL/Python etc. (though 100% important too), but to always examine each through the perspective of e.g. business / product angle. Think about the unique contribution, if any, to better understanding of the business, the product, of those engaged with it, and then, the actionable recommendations you may provide stakeholders as a data analyst.

Moreover, recall you may open a Substack account yourself to better understand the product if you feel that you got questions you don’t have answers for.

Here are some ideas:

1. Carefully review the Publications table. Identify the naming policy defect and craft the message how you’d report it.

2. Create a dashboard which provides visibility into:

   1. The total amount of users.

   2. The total amount of publications.

   3. The distribution of publications per publication category.

   4. How many subscribers there are & paid/free types.

   5. The distribution of subscribers & types per publication category.

   6. How many email-posts were sent.

3. Calculate and populate the open_rate (emailed_posts_stats table).

   1. Create a dashboard enabling to select a specific publication and compare its stats with the average of its peers (i.e. overall average and average of the publication category of the selected publication).

4. Who are the top 5% of users with the most subscribers? Free/Paid subscribers?

5. Are there publication categories which show more engagement (e.g. likes, comments, shares) by their readership compared to others?

6. Create a View for the current top-20 publications ranking system: global_rank, category_rank, publication_name, publicationID, publication_category (precise details of the company’s current methodology is in Option #1 above).

7. Ask yourself how the company generates revenue. Find out how Substack does it. For example, do the publishers pay it to host their e-newsletters (is it pay per X hosting-time, per post, et cetera), does it make money by collecting a fee off payments of paid subscribers or through online advertisements? And so on.

8. Conduct various cohorts tracking metrics & create visualizations / dashboards:

   1. e.g. see tracking publication cohorts and their posting engagement:

   2. track categories’ posting/subscriber(free/paid) etc. trends over time.

   3. track the engagement metric rates (likes, shares, comments) for posts cohorts. Add filtering per selected publication category to the dashboard.

9. Track the views:subscribers ratio for publications’ posts. Does it help you deduce any useful insights about the users’ ability to convert viewers into subscribers?

10. Review the Subscribers, Posts, Users, etc. tables. Explain what you would improve and why (It’s key to explain why it matters). Think tracking processes, changes.

11. Re-read the description of the Posts, Emailed_Posts_Stats, and Subscribers tables, paying close attention to the fields posts.new_free/paid_subscribers and emailed_posts_stats.recipients_number. Identify any gaps in how the data is being tracked.

    Use SQL to validate your hypothesis about the gap and craft a concise report to share with the data team. In your report:

    1. Clearly state the identified gap.

    2. Propose an engineering solution to address it.

    3. Suggest a data recovery plan for cases where past data cannot be retrieved. Include your SQL code for the recovery process and specify the table where the recovered data should be stored.

       1. side-note: Remember- data and software systems are inherently flexible. Basically anything that has been created, modified, or tracked can be adjusted after the fact. For instance:

          1. A user or publication can be created or deleted.

          2. Timestamps, such as "post_displayed_datetime," can theoretically be manipulated by users. e.g., to create the appearance of an older, "aged" publication for resale. Not a spoiler in this case ;) Though I thought about it ;)

12. Examine the email domains and their respective types. Evaluate if there are insights worth reporting.

13. Evaluate correlations between different metrics. Remember that correlation does not necessarily mean causation.

14. … Feel free to drop further interesting ideas in the comments / drop me a message via LinkedIn!

Share

Share novsto

Subscribe now

This is a lesson I learned as a Full Stack Product Analyst at Firefly.ai, where I was the Product Analyst, Data Product Manager and Data Engineer.

Faster results mean more focus and better analysis.

Here are two examples showcasing how to improve performance and cut costs without sacrificing data accuracy.

Subscribe now

Share

Connect on LinkedIn

APPLICATION EVENT LOGS:
Optimizing Event Log Processing

Scenario: You’re working with raw JSON data containing event logs. The goal is to transform this into a daily log of active users without vendor lock-in, ensuring cost-effective performance.1

Challenges:

• Large datasets (e.g., 10M+ rows).

• Avoiding redundant re-processing of rows.

• Avoiding vendor lock-in.

Solution:

• Add a timestamp column to the raw table (MiRRor). Every new record automatically includes its creation timestamp.

• Create a Point In Time (PIT) Metadata table (DIMensional) to store key processing details like the last processed date.

Benefits:

• No need to re-compute MAX(DATE) for every query (from FACTual table).

• Process only new rows added after the last run.

• Significantly faster processing and reduced compute costs.

How the procedure works:

1. Use PIT Metadata to check the last processed date.

2. Compare it with the timestamp of new records.

3. Parse only the unprocessed rows into a structured table for analysis.

Below, the process flow:

Illustrating further, and enabling anyone to test and compare efficiency, follow the process below:

CREATE DATABASE PIT_METADATA;
USE DATABASE PIT_METADATA;

download the dataset

CREATE TABLE PIT_METADATA.dbo.application_event_logs (event_logs NVARCHAR(MAX));

--loading 5 million out of total: 10,104,419 rows (event logs)
---LOAD CSV 
BULK INSERT PIT_METADATA.dbo.application_event_logs
FROM 'C:\Users\userName\Desktop\Documents\novsto\first_batch_(5m_event_logs).csv'
WITH (
FORMAT='CSV',
FIRSTROW=0
);

ALTER TABLE PIT_METADATA.dbo.application_event_logs
ADD TS_MS DATETIME2 NULL;

-- Add a DEFAULT constraint to insert the current system datetime for new rows
ALTER TABLE PIT_METADATA.dbo.application_event_logs
ADD CONSTRAINT DF_TS_MS_DEFAULT
DEFAULT (SYSDATETIME()) FOR TS_MS;

CREATE TABLE PIT_METADATA.dbo.application_event_logs_staging (event_logs NVARCHAR(MAX));

--loading remaining 5mln 104,410
BULK INSERT PIT_METADATA.dbo.application_event_logs_staging
FROM 'C:\Users\userName\Desktop\Documents\novsto\second_batch_(5_1m_event_logs).csv'
WITH (
    FORMAT = 'CSV',
    FIRSTROW = 1,
    FIELDTERMINATOR = ',',
    ROWTERMINATOR = '\n'
);

INSERT INTO PIT_METADATA.dbo.application_event_logs (event_logs)
SELECT event_logs
FROM PIT_METADATA.dbo.application_event_logs_staging;

DROP TABLE PIT_METADATA.dbo.application_event_logs_staging;

DYNAMIC COHORT POSTING ANALYSIS

Scenario: For an e-newsletter platform (e.g., Substack), you want to track monthly posting activity rates. The focus is on how user cohorts evolve over time, comparing their posting habits month-to-month.

Challenges:

• Dynamically handling growing data as new months are added.

• Calculating accurate rates with no manual updates.

Solution:

• Group users by cohorts (e.g., month of account creation).

• Use a dynamic query to calculate monthly posting rates:

  • Current Month Posts ÷ Previous Month Posts

  • Handle nulls and zero values to avoid errors.

• Save intermediate results in the PIT Metadata table for reuse.

• (not in this post: save rates calculations for closed months).

Benefits:

• No need to rebuild the entire dataset every time.

• Scales efficiently with growing data.

• Provides clear insights into user engagement trends.

*of course that further formatting such as color-coding for trends is great for yourself and your stakeholders to grasp trends faster (e.g. see top image of this post).

CREATE DATABASE eletters;
USE DATABASE eletters;

CREATE TABLE posts (
 publicationID	NVARCHAR(35) NOT NULL,
 postID	NVARCHAR(10) NOT NULL,
 createdAt DATETIME NOT NULL,
 post_title NVARCHAR(110),
 post_displayed_datetime DATETIME NOT NULL,
 views INT NOT NULL,
 new_free_subscribers INT NOT NULL,	
 new_paid_subscribers INT NOT NULL,
 likes INT NOT NULL,
 shares INT NOT NULL,
 comment_counter INT NOT NULL
);

CREATE TABLE publications (
 publicationID NVARCHAR(35) NOT NULL,
 publication_createdDate DATE NOT NULL,
 publication_name NVARCHAR(50) NOT NULL,
 publication_deletedDate DATE
  );

download Posts table data

download Publications table data

The code below generates all of the output shown above. It calculates it all from start to finish. It is more classic of what an analyst would usually do.

CREATE OR ALTER PROCEDURE DYNAMIC_RATES AS
BEGIN
    -- step0: declaring variables
    DECLARE @PIVOT_POST_DATE VARCHAR(MAX);
    DECLARE @NULL_TERMINATOR VARCHAR(MAX);
    DECLARE @QUERY NVARCHAR(MAX);

    -- step1: creating distinct YYYY-MM posting date groups
-- step2: saving the posting dates into two kind of string variables:
------2.1: PIVOT POST DATE is for the PIVOT function
------2.2: NULL TERMINATOR is to eliminate nulls in the columns (see distinction in output: "0.00" vs. "0")
    WITH distinct_posting_dates AS (
        SELECT DISTINCT FORMAT(createdAt, 'yyyy-MM') AS POST_DATE
        FROM posts
    )
    SELECT @PIVOT_POST_DATE = STRING_AGG('[' + POST_DATE + ']', ',') WITHIN GROUP (ORDER BY POST_DATE ASC),
        @NULL_TERMINATOR = STRING_AGG(', COALESCE([' + POST_DATE + '], ''0'') AS [' + POST_DATE + ']', '') WITHIN GROUP (ORDER BY POST_DATE ASC)
    FROM distinct_posting_dates;

    -- step3: building the dynamic query.
-------- note how the rates are calculated: 
-------- ensuring fraction ( * 1.00), 
-------- NULLIF to eliminate division by zero errors
-------- COALESCE to handle post-NULLification
-------- Rounding, turning into string for then to truncate fraction length
    SET @QUERY = '
    WITH posts_posted AS (
        SELECT publicationID,
               FORMAT(createdAt, ''yyyy-MM'') AS post_date,
               COUNT(postID) AS posts_amount
        FROM posts
        GROUP BY publicationID, FORMAT(createdAt, ''yyyy-MM'')
    )
    SELECT COHORT ' + @NULL_TERMINATOR + '
    FROM (
        SELECT 
            FORMAT(publication_CreatedDate, ''yyyy-MM'') AS COHORT,
            post_date AS POST_DATE,
            LEFT(
             CAST(
              ROUND(
               COALESCE(
                SUM(posts_amount) * 1.00
                  /
                NULLIF(
                 LAG(SUM(posts_amount)) OVER (
       PARTITION BY FORMAT(publication_CreatedDate, ''yyyy-MM'')
       ORDER BY FORMAT(publication_CreatedDate, ''yyyy-MM'')
             ), 0), 0), 2) AS VARCHAR(20)), 4
            ) AS POSTING_RATE
        FROM publications
        JOIN posts_posted
            ON publications.publicationID = posts_posted.publicationID
        GROUP BY FORMAT(publication_CreatedDate, ''yyyy-MM''), post_date
    ) AS sourceTable
    PIVOT (
        MAX(POSTING_RATE)
        FOR POST_DATE IN (' + @PIVOT_POST_DATE + ')
    ) AS pivoted_table
    ORDER BY COHORT ASC;
    ';

    -- step3: execute the dynamic query
    EXEC sp_executesql @QUERY;
END;

EXEC DYNAMIC_RATES;

This current one exemplifies that the PIT_METADATA table can be leveraged in another way compared to the first example. See the code below:

--create the table
CREATE TABLE eletters.dbo.PIT_METADATA (
	UDF_NAME NVARCHAR(1000),
	TABLE_NAME NVARCHAR(1000),
	VARIABLE_NAME NVARCHAR(1000),
	STRING_VALUE NVARCHAR(MAX),
	TIMESTAMP_VALUE DATETIME2(3),
	DATE_VALUE DATE,
	VALUE_LAST_UPDATED_AT DATETIME2(3)
	)
;
--create temporary table
CREATE TABLE #temp_metadata (
    PIVOT_POST_DATE NVARCHAR(MAX),
    NULL_TERMINATOR NVARCHAR(MAX)
);

--populate the table
WITH distinct_posting_dates AS (
    SELECT DISTINCT FORMAT(createdAt, 'yyyy-MM') AS POST_DATE
    FROM posts
    WHERE createdAt < '2021-06-01' --- so we can subsequently run and add the next date
)
-- Generate strings for both PIVOT_POST_DATE and NULL_TERMINATOR
INSERT INTO #temp_metadata (PIVOT_POST_DATE, NULL_TERMINATOR)
SELECT 
    STRING_AGG('[' + POST_DATE + ']', ',') WITHIN GROUP (ORDER BY POST_DATE ASC) AS PIVOT_POST_DATE,
    STRING_AGG(', COALESCE([' + POST_DATE + '], ''0'') AS [' + POST_DATE + ']', '') WITHIN GROUP (ORDER BY POST_DATE ASC) AS NULL_TERMINATOR
FROM distinct_posting_dates;

---insert values into PIT metadata
INSERT INTO eletters.dbo.PIT_METADATA (UDF_NAME, VARIABLE_NAME, STRING_VALUE, VALUE_LAST_UPDATED_AT)
SELECT 'DYNAMIC_RATES', 'PIVOT_POST_DATE', PIVOT_POST_DATE, '2021-05-30 10:10:10.111'
--SYSDATETIME() if it was in Production
FROM #temp_metadata;

INSERT INTO eletters.dbo.PIT_METADATA (UDF_NAME, VARIABLE_NAME, STRING_VALUE, VALUE_LAST_UPDATED_AT)
SELECT 'DYNAMIC_RATES', 'NULL_TERMINATOR', NULL_TERMINATOR, '2021-05-30 10:10:10.111'
--SYSDATETIME() if it was in Production
FROM #temp_metadata;

--drop the temporary table
DROP TABLE #temp_metadata;

CREATE OR ALTER PROCEDURE RECURRING_DYNAMIC_RATES AS
BEGIN
    --delcaring variables
    DECLARE @PIVOT_POST_DATE NVARCHAR(MAX);
    DECLARE @NULL_TERMINATOR NVARCHAR(MAX);
    DECLARE @QUERY NVARCHAR(MAX);
    DECLARE @LAST_UPDATED_AT DATETIME2(3);
--Current Date on Production should be SYSDATETIME(), i.e. not hardcoded like here.
    DECLARE @CURRENT_DATE DATETIME2(3) = CAST('2021-06-06 06:06:06.006' AS DATETIME2);

    --fetch metadata from PIT_METADATA table instead of re-computing it
    SELECT 
        @PIVOT_POST_DATE = STRING_VALUE,
        @LAST_UPDATED_AT = VALUE_LAST_UPDATED_AT
    FROM eletters.dbo.PIT_METADATA
    WHERE UDF_NAME = 'DYNAMIC_RATES'
      AND VARIABLE_NAME = 'PIVOT_POST_DATE';

    SELECT 
        @NULL_TERMINATOR = STRING_VALUE
    FROM eletters.dbo.PIT_METADATA
    WHERE UDF_NAME = 'DYNAMIC_RATES'
      AND VARIABLE_NAME = 'NULL_TERMINATOR';

    --check if metadata is outdated 
--(i.e. checks if need to update the strings because a new month started since last run)
    IF FORMAT(@CURRENT_DATE, 'yyyy-MM') > FORMAT(@LAST_UPDATED_AT, 'yyyy-MM')
    BEGIN
        -- append only the new month
------PIVOT POST DATE is for the PIVOT function
------NULL TERMINATOR is to eliminate nulls in the columns (see distinction in output: "0.00" vs. "0")
        SELECT @PIVOT_POST_DATE = @PIVOT_POST_DATE + ',[' + FORMAT(@CURRENT_DATE, 'yyyy-MM') + ']',
               @NULL_TERMINATOR = @NULL_TERMINATOR + ', COALESCE([' + FORMAT(@CURRENT_DATE, 'yyyy-MM') + '], ''0'') AS [' + FORMAT(@CURRENT_DATE, 'yyyy-MM') + ']'
        ;

        --update the metadata in PIT_METADATA table
        UPDATE eletters.dbo.PIT_METADATA
        SET STRING_VALUE = @PIVOT_POST_DATE,
            VALUE_LAST_UPDATED_AT = @CURRENT_DATE
        WHERE UDF_NAME = 'DYNAMIC_RATES'
          AND VARIABLE_NAME = 'PIVOT_POST_DATE';

        UPDATE eletters.dbo.PIT_METADATA
        SET STRING_VALUE = @NULL_TERMINATOR,
            VALUE_LAST_UPDATED_AT = @CURRENT_DATE
        WHERE UDF_NAME = 'DYNAMIC_RATES'
          AND VARIABLE_NAME = 'NULL_TERMINATOR';
    END

    -- the dynamic query:
-------- note how the rates are calculated: 
-------- ensuring fraction ( * 1.00), 
-------- NULLIF to eliminate division by zero errors
-------- COALESCE to handle post-NULLification
-------- Rounding, turning into string for then to truncate fraction length
    SET @QUERY = '
    WITH posts_posted AS (
        SELECT publicationID,
               FORMAT(createdAt, ''yyyy-MM'') AS post_date,
               COUNT(postID) AS posts_amount
        FROM posts
        GROUP BY publicationID, FORMAT(createdAt, ''yyyy-MM'')
    )
    SELECT COHORT ' + @NULL_TERMINATOR + '
    FROM (
        SELECT 
            FORMAT(publication_CreatedDate, ''yyyy-MM'') AS COHORT,
            post_date AS POST_DATE,
            LEFT(
                CAST(
                    ROUND(
                        COALESCE(
                            SUM(posts_amount) * 1.00
                            / NULLIF(LAG(SUM(posts_amount)) OVER (
                                PARTITION BY FORMAT(publication_CreatedDate, ''yyyy-MM'')
                                ORDER BY FORMAT(publication_CreatedDate, ''yyyy-MM'')
                            ), 0), 0
                        ),
                        2
                    ) AS VARCHAR(20)
                ), 4
            ) AS POSTING_RATE
        FROM publications
        JOIN posts_posted
            ON publications.publicationID = posts_posted.publicationID
        GROUP BY FORMAT(publication_CreatedDate, ''yyyy-MM''), post_date
    ) AS sourceTable
    PIVOT (
        MAX(POSTING_RATE)
        FOR POST_DATE IN (' + @PIVOT_POST_DATE + ')
    ) AS pivoted_table
    ORDER BY COHORT ASC;
    ';

    -- execute the dynamic query
    EXEC sp_executesql @QUERY;
END;

EXEC RECURRING_DYNAMIC_RATES;

Moreover, the saving of the rates-data itself isn’t addressed in this particular code example (though definitely worth it). Yet that’s definitely something you’d want to coordinate with your manager / data engineers.

Share

Subscribe now

Connect on LinkedIn

Key Takeaways:

• Efficiency Matters: Streamlining processes saves time and costs, improving overall performance.

• Leverage Metadata: PIT tables store reusable values, reducing redundant computations.

• Think Dynamically: Design solutions that adapt to growing data without manual intervention.

By applying these techniques, you’ll not only save costs but also make your workflows more scalable and future-proof. Start small, test often, and refine your approach. Efficiency is the key to unlocking better insights.

That said, coordinate such efforts with your manager and the data engineering.

We’ll try to convince him using data!

In this piece I briefly describe how I've created a Tableau dashboard using alarms data of Pikud HaOref to help people.

So find out when you are least/most likely to be disturbed by rocket alerts here:

https://public.tableau.com/views/alarmsatIsrael/Dashboard1?:language=en-US&:display_count=n&:origin=viz_share_link

Subscribe now

Share

Getting the data

The source of the data was Pikud HaOref’s website. So this data is official data.

Obtain the data effortlessly using this link, adjust the dates as you please, and save as JSON files:

https://www.oref.org.il//Shared/Ajax/GetAlarmsHistory.aspx?lang=en&fromDate=07.10.2023&toDate=28.10.2023&mode=0

Note that the data is limited to 2,000 cases per calendar day, which is a possible weakness one has to pay attention to. In this case, it was not an issue since I had no plan to include the first day of the war (October 7th) as it represented an extreme day which was far from representing subsequent days of war-routine.

The Excel Part

Load each JSON file, run it through the quick process of instructing Excel to re-organize/structure the data from JSON format to tabular data.

Iterate per JSON file that you have.

Time to Append the data (i.e. similar to UNION ALL in SQL).

*Please ensure you have the same amount of columns and in the same order. If you play around with it, you might not be able to properly append the data.

Alternatively, you may append the tables by “standing” on one of the fields in one of the data tables. Then do as in the image below:

Now it as if we’ve stacked the data tables one of top of each other, similar to SQL Union All. But in SQL we’d use UNION to ensure we have unique rows only. Proceed as in the image below to remove duplicate rows:

Now I merely recommend to assign the proper fields format per column in Excel, particularly if you are using the free version of Tableau.

The Tableau Part

I explain how I handled key parts in Tableau to create this dashboard. Tableau is ever-evolving and it’s out of scope to teach it from scratch.
I recommend beginners to do a course and then practice with ChatGPT instructions instead of frustrating yourself/wasting time in figuring out how to do stuff.

• I chose which visualizations to use, choice of style, colors, et cetera, based on a summary of a book I’ve read (storytelling with data: a data visualization guide for business professionals). I recommend Cole Nussbaumer Knaflic’s website. She even shares some Excel template files!

Filter

Regardless if you work on the Tableau Desktop or via the browser, set the scope of the data, and ensure it is connected/linked or disconnected/not linked as appropriate in adjusting the dashboard, per visualization.

In this case, I needed all visualizations to be connected and have the same scope.

• Excluded October 7th data (extreme outlier).

• Category Desc set only for ‘Missiles’.

• Location to include all as-is. Adjustment to happen only on the dashboard itself.

Calculated Fields

(1) “5minIntervals”: Since Tableau’s CASE WHEN is different from what I’m used to, I opted for the following code in the calculated field:

IF DATEPART('minute', [Alert Date]) >= 0 
 AND DATEPART('minute', [Alert Date]) < 5 THEN '00-04'
ELSEIF DATEPART('minute', [Alert Date]) >= 5 
 AND DATEPART('minute', [Alert Date]) < 10 THEN '05-09'
ELSEIF DATEPART('minute', [Alert Date]) >= 10 
 AND DATEPART('minute', [Alert Date]) < 15 THEN '10-14'
ELSEIF DATEPART('minute', [Alert Date]) >= 15 
 AND DATEPART('minute', [Alert Date]) < 20 THEN '15-19'
ELSEIF DATEPART('minute', [Alert Date]) >= 20 
 AND DATEPART('minute', [Alert Date]) < 25 THEN '20-24'
ELSEIF DATEPART('minute', [Alert Date]) >= 25 
 AND DATEPART('minute', [Alert Date]) < 30 THEN '25-29'
ELSEIF DATEPART('minute', [Alert Date]) >= 30 
 AND DATEPART('minute', [Alert Date]) < 35 THEN '30-34'
ELSEIF DATEPART('minute', [Alert Date]) >= 35 
 AND DATEPART('minute', [Alert Date]) < 40 THEN '35-39'
ELSEIF DATEPART('minute', [Alert Date]) >= 40 
 AND DATEPART('minute', [Alert Date]) < 45 THEN '40-44'
ELSEIF DATEPART('minute', [Alert Date]) >= 45 
 AND DATEPART('minute', [Alert Date]) < 50 THEN '45-49'
ELSEIF DATEPART('minute', [Alert Date]) >= 50 
 AND DATEPART('minute', [Alert Date]) < 55 THEN '50-54'
ELSE '55-59'
END

(2) Formatted Hour of Alert: (Hours alerts happened - השעות בהן היו אזעקות)

STR(DATEPART('hour', [Alert Date])) + 
IF DATEPART('hour', [Alert Date]) < 12 THEN " AM" ELSE " PM" END

(3) Choosing the right filter, showing legends, and choosing proper titles is key.
I’m not saying mine are perfect, but it’s always better to invest time into thinking about those stuff than cracking how to do something on your own instead of getting guidance from ChatGPT etc. Keep it in mind.

Subscribe now

Share

Share novsto

Feel free to connect via LinkedIn :)

• The dashboard

• Please keep in mind that the terrorists may change their modus operandi and then the dashboard’s assessment would no longer be relevant.

Key points:

• Approach the data analytically. I recommend starting vertically (per variable/column), then horizontally (per row), and then looking at the entire story.

• Be verbal- Clarifications & assumptions. Explain your methodology, thoughts.
  It’s a good opportunity to show creativity and critical thinking skills.

• The data itself is neither good nor bad- Compose good and bad story/stories per set of data. Then decide on your final answer if the account has been taken over or not, degree of confidence, et cetera (as the format may change per interviewer).

• Think outside the box- Consider what would you say in case you’d be asked, for example, “what other data/variables would’ve helped you assess the situation?”,
  or “what manipulations/calculations to the existing data/variables would assist you in making a better assessment?”

Share

Side note: For an introduction to the world of fraud detection and prevention, I highly recommend the book of Gilit Saporta and Shoshana Maraney, “Practical Fraud Prevention: Fraud and AML Analytics for FinTech and eCommerce Using SQL and Python” (O’Reilly, 2022). The book also has a chapter dealing with the subject of Account Takeovers.

Note that you can create for yourself additional practice sets precisely like I did here:
(1) I got data samples, (2) structured some scenarios to teach ChatGPT, and then (3) instructed it to generate new cases for me to practice. Make sure you instruct it to get creative in building a stories behind each case to facilitate better results.

If done face to face, or given space to leave text, you may ask for clarifications / note your assumptions. In this case, let’s say you may assume:

• All records refer to bank/financial institutions accounts.

• All of the records start from the first login of the users to their accounts.

• Each case is for one user, one account, and no account has more than one user.

• LoginDateTime was localized (see Bonus part); format: MM/DD/YYYY HH:MM.

Case Study #1: Tina

Case Study #2: Alice

Case Study #3: Osher

My approach and some thoughts:

In general, I recommend - particularly when given to do it on Excel, paper, or white board - to start vertically, then horizontally, then entire case.

• Vertically: Understand the variables & mark any unique part in the data itself
  (e.g. the tiny change in user agent because later it’s easier to think, for instance, if a tiny change in user agent is possibly an upgrade of same device or different device, etc.).

• Horizontally, row by row, begin seeing how the combination of the different variables connect with one another and what you can make of it. That’s a good time to think of manipulations/calculations with minimal biases and heuristics.

• Evaluate the story as a whole to reach the final decision.

Case Study #1: Tina -

Bottom-line: Good story triumphed over bad story.

I imagined Tina traveled with her laptop (user agent showed it was the same device throughout the records), and the login pattern seemed consistent in terms of 1 login per day at around same time frame, except for when she enjoyed her vacation.

Case Study #2: Alice -

Bottom-line: Bad story triumphed over good story. Row 8 was when she had her details exposed (her device was probably hacked too), suspecting row 13 was the fraudster logged into her account to sniff around.

I imagined Alice owning two devices, iPhone and iPad, which she routinely used from the same place (always connection from the same place).

Her routine: daily account-check, aside from when she wanted to install the official application on a new device. She then seemed to first, log into her account via web, and only then used the direct link to get to download, install, and log into the bank’s official application that she used thereon.

The 7 days of the week was also an indicator worth keeping in mind when I realized I was looking at consecutive days. In Alice’s case, the logins’ hour/time of the day had a pattern relevant to whether it was a work day or weekend. This was much easier to see and verify using SQL (though if such a test is on a white board or paper - just say it!).

SELECT CASE WHEN DATENAME(DW,LoginDateTime) IN ('Saturday', 'Sunday') 
 THEN 'weekend'
 ELSE 'workday'
 END AS isWeekend
 , DATENAME(DW,LoginDateTime) AS [DayName]
 , FORMAT(LoginDateTime, 'HH:mm') AS [HourMinute]
 , *
FROM ATO
WHERE userID = 'N26'

Note the “LastSiteVisited” on row 8 provided a strong indication that the fraudster used an overlay as the modus operandi. This cue helped interpret the rest of the data too.

Moreover, I suspected row 13 was the fraudster logged into the account because it imitated Alice’s activity and data. But it didn’t fit in her routine: not using application once installed on device; 2nd login that day; hour would fit during the week (when account was compromised), not weekend.

Furthermore, while I believe faking device data is possible, I don’t think it’s possible to fake IP. To the best of my knowledge, it’d require access to the network. That was why I thought her device/network were hacked and not just the credentials stolen.

Case Study #3: Osher -

Bottom-line: Good story triumphed over bad story. Always lean towards the good story unless the bad story overwhelmingly beats the good/plausible story. In short, no clear indication of ATO or strong suspicion of one.

Osher checked the account once per 3 days from a number of different devices (1 phone, 1 tablet, 3 computers). 3 computers made me imagine some high tech employee with his own personal computer, work computer and possibly old PC or spouse’s or so computer. Hybrid work. It may also mean one of the different locations and IPs result from logging in from the company office (likely earlier logins from his own house due to the variety of devices).

With the exception of two times, he usually logged in around lunch break. Yet I see no reason to suspect the deviation from this pattern when it was around the same location, pattern of 1 check per 3 days remained, same way to login, and device was not new.

Last login could have many good explanations as well (2nd office, change of apartment, change of ISP of same apartment, etc.).

SELECT userID
 , FORMAT(LoginDateTime, 'yyyy-MM-dd HH:mm') AS LoginDateTime
 , CASE WHEN DATENAME(DW,LoginDateTime) IN ('Friday', 'Saturday') 
  THEN 'weekend'
  ELSE 'workday'
  END AS isWeekend
 , DATENAME(DW,LoginDateTime) AS [DayName]
 , FORMAT(LoginDateTime, 'HH:mm') AS [HourMinute]
 , COALESCE(
   DATEDIFF(DAY, 
   LAG(LoginDateTime) OVER (ORDER BY LoginDateTime ASC)
   , LoginDateTime)
   ,0) AS DaysBetweenLogins
 , COALESCE(
   DATEDIFF(HOUR, 
   LAG(LoginDateTime) OVER (ORDER BY LoginDateTime)
   , LoginDateTime)
   ,0) AS HoursBetweenLogins
 , Device
 , DeviceGroup
 , Browser
 , OS
 , UserAgent
 , [IP]
 , ISP
 , CountryLevel
 , StateRegionLevel
 , CityLevel
 , LastSiteVisited
FROM ATO
WHERE userID = 'N31'

Thinking outside of the box: What additional data could be useful here? Perhaps…

• VPN used- Boolean

• Name of network connected from

• Amount of time logged into the account (and calculate average)

• Pages viewed inside account / number of pages viewed

• Any ‘biometric-behavior’ data of the user

Subscribe now

Bonus

Built in time zone converters are smarter and easier to use, and it is worth to know such options exist, particularly when trying to immerse yourself in the case, finding the anomaly and indicators someone other than the rightful owner accessed the account.

AT TIME ZONE is a feature in T-SQL that allows for the conversion of datetime values between different time zones. It automatically accounts for Daylight Saving Time adjustments based on the specified time zone and the date of the datetime value.

Use SELECT * FROM sys.time_zone_info to know the exact time zone names SQL Server uses.

SELECT userID
, CountryLevel
, StateRegionLevel
, CASE 
-- Convert from Israel time zone to UTC
WHEN CountryLevel = 'Israel' THEN 
CONVERT(DATETIME, LoginDateTimeLocalized AT TIME ZONE 'Israel Standard Time' AT TIME ZONE 'UTC')
-- Convert from USA - California time zone to UTC
WHEN CountryLevel = 'USA' AND StateRegionLevel = 'California' THEN 
CONVERT(DATETIME, LoginDateTimeLocalized AT TIME ZONE 'Pacific Standard Time' AT TIME ZONE 'UTC')
-- Convert from USA - Hawaii time zone to UTC
WHEN CountryLevel = 'USA' AND StateRegionLevel = 'Hawaii' THEN 
CONVERT(DATETIME, LoginDateTimeLocalized AT TIME ZONE 'Hawaiian Standard Time' AT TIME ZONE 'UTC')
-- Convert from USA - Hawaii time zone to UTC
WHEN CountryLevel = 'USA' AND StateRegionLevel = 'Maryland' THEN 
CONVERT(DATETIME, LoginDateTimeLocalized AT TIME ZONE 'Eastern Standard Time' AT TIME ZONE 'UTC')
-- Add more conditions for other states/regions as needed
ELSE null -- Default behavior (quick indicator of missing combination)
END AS LoginDateTimeUTC
, LoginDateTimeLocalized
FROM step1

Create ATO table & case studies data

If you want to run the SQL code on your own.

CREATE TABLE ATO (
    userID NVARCHAR(50) NOT NULL,
    LoginDateTime DATETIME NOT NULL,
    Device NVARCHAR(100),
    DeviceGroup NVARCHAR(100),
    Browser NVARCHAR(100),
    OS NVARCHAR(100),
    UserAgent NVARCHAR(MAX),
    IP NVARCHAR(50),
    ISP NVARCHAR(100),
    CountryLevel NVARCHAR(100),
    StateRegionLevel NVARCHAR(100),
    CityLevel NVARCHAR(100),
    LastSiteVisited NVARCHAR(255)
);

-- Alice data
INSERT INTO ATO (userID, LoginDateTime, Device, DeviceGroup, Browser, OS, UserAgent, IP, ISP, CountryLevel, StateRegionLevel, CityLevel, LastSiteVisited) VALUES ('N26', '10/01/2021  18:30:00 PM', 'Apple iPhone', 'Smart Phone', 'Safari', 'iOS', 'Mozilla/5.0 (iPhone; CPU iPhone OS 13_2_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.3 Mobile/15E148 Safari/604.1', '203.0.113.45', 'Verizon', 'USA', 'Maryland', 'Baltimore', 'https://www.google.com');
INSERT INTO alice_ATO (userID, LoginDateTime, Device, DeviceGroup, Browser, OS, UserAgent, IP, ISP, CountryLevel, StateRegionLevel, CityLevel, LastSiteVisited) VALUES ('N26', '10/01/2021  18:45:51 PM', 'Apple iPhone', 'Smart Phone', NULL, 'iOS', 'AppleApp_bankOfAmerica/23.10 (iPhone; CPU iPhone OS 13_2_3 like Mac OS X)', '203.0.113.45', 'Verizon', 'USA', 'Maryland', 'Baltimore', NULL);
INSERT INTO alice_ATO (userID, LoginDateTime, Device, DeviceGroup, Browser, OS, UserAgent, IP, ISP, CountryLevel, StateRegionLevel, CityLevel, LastSiteVisited) VALUES ('N26', '10/02/2021  14:01:11 PM', 'Apple iPhone', 'Smart Phone', NULL, 'iOS', 'AppleApp_bankOfAmerica/23.10 (iPhone; CPU iPhone OS 13_2_3 like Mac OS X)', '203.0.113.45', 'Verizon', 'USA', 'Maryland', 'Baltimore', NULL);
INSERT INTO alice_ATO (userID, LoginDateTime, Device, DeviceGroup, Browser, OS, UserAgent, IP, ISP, CountryLevel, StateRegionLevel, CityLevel, LastSiteVisited) VALUES ('N26', '10/03/2021  14:05:22 PM', 'Apple iPhone', 'Smart Phone', NULL, 'iOS', 'AppleApp_bankOfAmerica/23.10 (iPhone; CPU iPhone OS 13_2_3 like Mac OS X)', '203.0.113.45', 'Verizon', 'USA', 'Maryland', 'Baltimore', NULL);
INSERT INTO alice_ATO (userID, LoginDateTime, Device, DeviceGroup, Browser, OS, UserAgent, IP, ISP, CountryLevel, StateRegionLevel, CityLevel, LastSiteVisited) VALUES ('N26', '10/04/2021  18:12:34 PM', 'Apple iPhone', 'Smart Phone', NULL, 'iOS', 'AppleApp_bankOfAmerica/23.10 (iPhone; CPU iPhone OS 13_2_3 like Mac OS X)', '203.0.113.45', 'Verizon', 'USA', 'Maryland', 'Baltimore', NULL);
INSERT INTO alice_ATO (userID, LoginDateTime, Device, DeviceGroup, Browser, OS, UserAgent, IP, ISP, CountryLevel, StateRegionLevel, CityLevel, LastSiteVisited) VALUES ('N26', '10/05/2021  18:07:18 PM', 'Apple iPhone', 'Smart Phone', NULL, 'iOS', 'AppleApp_bankOfAmerica/23.10 (iPhone; CPU iPhone OS 13_2_3 like Mac OS X)', '203.0.113.45', 'Verizon', 'USA', 'Maryland', 'Baltimore', NULL);
INSERT INTO alice_ATO (userID, LoginDateTime, Device, DeviceGroup, Browser, OS, UserAgent, IP, ISP, CountryLevel, StateRegionLevel, CityLevel, LastSiteVisited) VALUES ('N26', '10/06/2021  18:40:18 PM', 'Apple iPad', 'Tablet and iPad', 'Chrome', 'iOS', 'Mozilla/5.0 (iPad; CPU OS 12_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/76.0.3809.81 Mobile/15E148 Safari/605.1', '203.0.113.45', 'Verizon', 'USA', 'Maryland', 'Baltimore', 'https://www.bankofameri.ca/redirecter');
INSERT INTO alice_ATO (userID, LoginDateTime, Device, DeviceGroup, Browser, OS, UserAgent, IP, ISP, CountryLevel, StateRegionLevel, CityLevel, LastSiteVisited) VALUES ('N26', '10/06/2021  18:43:37 PM', 'Apple iPad', 'Tablet and iPad', NULL, 'iOS', 'AppleApp_bankOfAmerica/23.10 (iPad; CPU OS 12_3 like Mac OS X)', '203.0.113.45', 'Verizon', 'USA', 'Maryland', 'Baltimore', NULL);
INSERT INTO alice_ATO (userID, LoginDateTime, Device, DeviceGroup, Browser, OS, UserAgent, IP, ISP, CountryLevel, StateRegionLevel, CityLevel, LastSiteVisited) VALUES ('N26', '10/07/2021  18:06:06 PM', 'Apple iPad', 'Tablet and iPad', NULL, 'iOS', 'AppleApp_bankOfAmerica/23.10 (iPad; CPU OS 12_3 like Mac OS X)', '203.0.113.45', 'Verizon', 'USA', 'Maryland', 'Baltimore', NULL);
INSERT INTO alice_ATO (userID, LoginDateTime, Device, DeviceGroup, Browser, OS, UserAgent, IP, ISP, CountryLevel, StateRegionLevel, CityLevel, LastSiteVisited) VALUES ('N26', '10/08/2021  18:20:16 PM', 'Apple iPad', 'Tablet and iPad', NULL, 'iOS', 'AppleApp_bankOfAmerica/23.10 (iPad; CPU OS 12_3 like Mac OS X)', '203.0.113.45', 'Verizon', 'USA', 'Maryland', 'Baltimore', NULL);
INSERT INTO alice_ATO (userID, LoginDateTime, Device, DeviceGroup, Browser, OS, UserAgent, IP, ISP, CountryLevel, StateRegionLevel, CityLevel, LastSiteVisited) VALUES ('N26', '10/09/2021  14:14:14 PM', 'Apple iPad', 'Tablet and iPad', NULL, 'iOS', 'AppleApp_bankOfAmerica/23.10 (iPad; CPU OS 12_3 like Mac OS X)', '203.0.113.45', 'Verizon', 'USA', 'Maryland', 'Baltimore', NULL);
INSERT INTO alice_ATO (userID, LoginDateTime, Device, DeviceGroup, Browser, OS, UserAgent, IP, ISP, CountryLevel, StateRegionLevel, CityLevel, LastSiteVisited) VALUES ('N26', '10/09/2021  19:31:01 PM', 'Apple iPad', 'Tablet and iPad', 'Chrome', 'iOS', 'Mozilla/5.0 (iPad; CPU OS 12_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/76.0.3809.81 Mobile/15E148 Safari/605.1', '203.0.118.45', 'Verizon', 'USA', 'Maryland', 'Baltimore', 'https://www.google.com');

--- Osher data
INSERT INTO ATO (userID, LoginDateTime, Device, DeviceGroup, Browser, OS, UserAgent, IP, ISP, CountryLevel, StateRegionLevel, CityLevel, LastSiteVisited) VALUES 
('N31', '9/2/2023 13:27', 'iPhone', 'iPhone', 'Safari', 'iOS', 'Mozilla/5.0 (iPhone; CPU iPhone OS 13_2_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.3 Mobile/15E148 Safari/604.1', '212.143.19.183', 'Bezeq', 'Israel', 'Tel Aviv', 'Tel Aviv', 'https://www.google.com'),
('N31', '9/5/2023 14:24', 'Desktop', 'Desktop', 'Safari', 'Mac', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.4 Safari/605.1.15', '212.143.19.183', 'Bezeq', 'Israel', 'Tel Aviv', 'Tel Aviv', 'https://www.google.com'),
('N31', '9/8/2023 13:16', 'Desktop', 'Desktop', 'Chrome', 'Windows', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36', '212.143.19.183', 'Bezeq', 'Israel', 'Tel Aviv', 'Tel Aviv', 'https://www.google.com'),
('N31', '9/11/2023 14:30', 'Samsung Galaxy Tab S4', 'Tablet', 'Other', 'Android', 'Mozilla/5.0 (Linux; Android 9; SM-T830 Build/PPR1.180610.011; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/76.0.3809.89 Safari/537.36', '212.143.19.184', 'Bezeq', 'Israel', 'Tel Aviv', 'Tel Aviv', 'https://www.google.com'),
('N31', '9/14/2023 14:35', 'Desktop', 'Desktop', 'Safari', 'Mac', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.4 Safari/605.1.15', '212.143.1.100', 'Bezeq', 'Israel', 'Tel Aviv', 'Tel Aviv', 'https://www.google.com'),
('N31', '9/17/2023 22:35', 'iPhone', 'iPhone', 'Safari', 'iOS', 'Mozilla/5.0 (iPhone; CPU iPhone OS 13_2_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.3 Mobile/15E148 Safari/604.1', '212.143.19.184', 'Bezeq', 'Israel', 'Tel Aviv', 'Tel Aviv', 'https://www.google.com'),
('N31', '9/20/2023 19:20', 'iPhone', 'iPhone', 'Safari', 'iOS', 'Mozilla/5.0 (iPhone; CPU iPhone OS 13_2_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.3 Mobile/15E148 Safari/604.1', '909.303.1.505', 'Cellcom', 'Israel', 'Tel Aviv', 'Ramat Gan', 'https://www.google.com'),
('N31', '9/23/2023 13:16', 'Desktop', 'Desktop', 'Safari', 'Mac', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.4 Safari/605.1.15', '212.143.19.184', 'Bezeq', 'Israel', 'Tel Aviv', 'Tel Aviv', 'https://www.google.com'),
('N31', '9/26/2023 14:14', 'Desktop', 'Desktop', 'Safari', 'Mac', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.4 Safari/605.1.15', '909.303.17.777', 'Cellcom', 'Israel', 'Tel Aviv', 'Tel Aviv', 'https://www.google.com');

Share

Subscribe now

Share novsto

1. Technically:

   1. Excel: Get Data From file → JSON

   2. SQL: INTERSECT; REPLACE(); LOWER()

2. Story Summary:
   December 2022, we experienced another round of layoffs at the startup I used to work in. Empty space. Sad atmosphere. We wanted to create a mutually loved playlist to be listened to for all of the remaining team members to sing along to. This meant for both, YouTube Music and Spotify users.
   So I took it on myself to run a MVP style project. I created the ultimate, the grooviest, of shared office space all-liked playlist! (well, with English bias…). Something quick and dirty to raise our spirits.
   This is a re-creation of what I’ve done back then.

Quick introduction to INTERSECT (and how it’s different to UNION)

INTERSECT returns the common records that exist in both of the result sets produced by two or more SELECT queries. The resulting dataset only includes unique rows that appear in the tables, and the column structures must match.
Importantly, INTERSECT happens after the SELECT in terms of order of operations.

For example, if you have two tables listing songs liked by different employees, you can use INTERSECT to quickly find songs that are liked by both, helping you curate a mutual playlist for the office.

Let's clarify how INTERSECT is different to UNION:
UNION combines the result sets of two or more SELECT queries into a single result set, eliminating duplicate rows. Unlike INTERSECT, which only returns rows common to each table, UNION returns all unique rows from both tables.

For example, if you have two lists of songs names from different albums, using UNION would give you a single list containing all unique songs across both albums.

Quick illustration for folks who need less abstract and more visualized example:

Share

Google Takeout: Get your YouTube Music data

Do a quick googling and you’ll find Google Takeout where you are easily instructed how to request to have your data prepared for you to download. Make sure you choose to get it in CSV format and choose just what’s relevant for you.

We’ll be working with the “music-library-songs” csv.

Google prepares this data fast. You’ll soon enough have it ready. Just download it.

Get your Spotify Music data

You have to be a premium Spotify user for getting the desired “yourLibrary” data file (all your liked songs, albums, playlists, etc.).

As I am no longer one, in here I re-read how the structure of their JSON is and instructed ChatGPT to generate data accordingly (though I told it to go wild and create input as it pleases when it comes to songs etc.).

But making a request to get the data from Spotify is pretty straightforward:

• You go online to their website www.spotify.com →

• login to your user →

• Privacy Settings →

• There you have a sub section called “Download your data.”

The data is not ready as fast as Google sends it but you will not be waiting long before it gets to you.

Note: as you may notice, there will likely be a bias against non-English texts. Hebrew, for instance, often ends up as a weird gibberish style text.

Excel: Prepare the Spotify data from JSON to CSV

Although it’s possible to load into Microsoft SSMS the JSON files right away, I’ve found it more useful to generally know how to handle JSON files using Excel.

Data → Get Data → From File → From JSON

Click the green text to dig deeper into the data. In the below image, it’s “Record”.

Choose each time which you’d like to expand as their structures are different. Later combine. Also, the Convert Into Table is a useful feature that needs clicking on to allow multi-expansions of arrays/objects.

Clicking the Convert to Table, you can expand each Record within likedSongs.

You may change the name of the first column to easily identify the source being expanded. Note the difference in signs which columns can be expanded and which cannot, as shown in the image.

Then click the Close & Load button.

After I finished with linkedSongs, I did the same for the likedAlbums and likedPlaylists due to their different structures. I chose to open only the relevant columns/variables.

I then combined them all into one table. If there were many rows, it would’ve been easier to import each to SQL using INSERT BULK and then using UNION.

That’s it for the Spotify data, taken from JSON file into Excel. I later converted it into CSV as I imported it to Microsoft’s SSMS (SQL Server Management Studio).

*For the record, I did not run due diligence on the correctness of the data that ChatGPT created.

Share

Create & insert the data

You can download the data I used for this example:

• “myChatGPTlikedSpotifyLibrary” JSON
  (or, in case you want to skip the JSON & Excel part, click here for the spotify_liked_library CSV);

• “myRedactedYouTubeMusic” CSV.

… or even better, get from Google & Spotify your own data with your friends & colleagues. Enjoy some tunes together :)

CREATE DATABASE mutual_office_music;
GO

USE mutual_office_music;
GO
-------------------------------------
--drop table spotify_music_emp1
CREATE TABLE spotify_music_emp1 (
    artistName NVARCHAR(255),
    trackName NVARCHAR(255),
    albumName NVARCHAR(255)
);
GO

BULK INSERT spotify_music_emp1
FROM 'C:\Users\....\Desktop\spotify_liked_library.csv' 
WITH (
    FIRSTROW = 2,
    FIELDTERMINATOR = ',', 
    ROWTERMINATOR = '\n',
    TABLOCK
);
GO
---
--drop table YouTube_music_emp2
--pay attention that you might need to remove the space between
--the words (variables/columns) if you've downloaded your own data
-- or alternatively, use [Song URL] etc. when you create the tables.
CREATE TABLE YouTube_music_emp2 (
    SongURL NVARCHAR(MAX),
    SongTitle NVARCHAR(MAX),
    AlbumTitle NVARCHAR(MAX),
    ArtistNames NVARCHAR(MAX)
);
GO

BULK INSERT YouTube_music_emp2
FROM 'C:\Users\....\Desktop\music-library-songs.csv' --name of YouTubeMusicFile
WITH (
    FIRSTROW = 2,
    FIELDTERMINATOR = ',', 
    ROWTERMINATOR = '\n',
    TABLOCK
);
GO

Subscribe now

Creating the shared playlist with SQL

The order of operations means we can first clean a bit the data to try and increase matches (i.e. the SELECT clause runs first and after it the INTERSECT).

As far as I can remember, I quickly glanced at the data and removed some capitalization variances as well as: ‘, commas, dots, question, exclamation and quotation marks in the data which could easily be cleaned out.

However, I did not handle the gibberish created by non-English/Latin letters and the diacritic marks (e.g. ä, ö, ü, å). The idea was to invest minimal time into this playlist & we had non-Hebrew folks at the office too, so a bias against Hebrew songs was actually welcomed.1

Originally, I’ve had about 5-7 different files provided by different employees, but the concept is the same.

SELECT [artist names] = REPLACE(
	REPLACE(
	REPLACE(
	REPLACE(
	REPLACE(
	REPLACE(
	LOWER(ArtistNames)
	, '''','')
	,',','')
	,'.','')
	,'?','')
	,'!','')
	,'"','')
, [song name] = REPLACE(
	REPLACE(
	REPLACE(
	REPLACE(
	REPLACE(
	REPLACE(
	LOWER(SongTitle)
	, '''','')
	,',','')
	,'.','')
	,'?','')
	,'!','')
	,'"','')
, [album name] = REPLACE(
	REPLACE(
	REPLACE(
	REPLACE(
	REPLACE(
	REPLACE(
	LOWER(AlbumTitle)
	, '''','')
	,',','')
	,'.','')
	,'?','')
	,'!','')
	,'"','')
FROM YouTube_music_emp2
INTERSECT
SELECT [artist names] = REPLACE(
	REPLACE(
	REPLACE(
	REPLACE(
	REPLACE(
	REPLACE(
	LOWER(artistName)
	, '''','')
	,',','')
	,'.','')
	,'?','')
	,'!','')
	,'"','')
, [song name] = REPLACE(
	REPLACE(
	REPLACE(
	REPLACE(
	REPLACE(
	REPLACE(
	LOWER(trackName)
	, '''','')
	,',','')
	,'.','')
	,'?','')
	,'!','')
	,'"','')
, [album name] = REPLACE(
	REPLACE(
	REPLACE(
	REPLACE(
	REPLACE(
	REPLACE(
	LOWER(albumName)
	, '''','')
	,',','')
	,'.','')
	,'?','')
	,'!','')
	,'"','')
FROM spotify_music_emp1

The output based on the example set:

Share

Share novsto

Subscribe now

I hope you enjoyed from this quick introduction to INTERSECT.

Feel free to comment and to connect :)

Connect on LinkedIn

The Tic Tac Toe SQL Challenge

The challenge is to write SQL code which can indicate X’s next move - the winning move. This is for any combination on the table - horizontal, vertical or diagonal.

Gist

Some of the stuff you may expect to find in two optional solutions below:

Solution #1 uses:

• CTE; Subquery; CONCAT; UNION.

It’s the simplest, most basic solution. Probably something one could be given to write during a job interview under limited time. Feel free to give yourself some 20-30 minutes to solve it if you wish to practice solving SQL question under time pressure.

Solution #2 uses:

• ISNULL(); STRING_AGG(/group_concat); GROUP WITHIN; CHARINDEX; GROUP BY.

What is...?
* CHARINDEX: The CHARINDEX function searches for a substring inside a string and returns the starting position of the substring if it is found. If the substring is not found, it returns 0.

Syntax:
CHARINDEX ( substring , string [ , start_location ] )

- substring: The substring to find.
- string: The string to search within.
- start_location: Optional. The position to start searching for the substring within string.

Example: SELECT CHARINDEX('cat', 'concatenation');  -- Output: 4

What is...?
* STRING_AGG: In Transact-SQL (Microsoft SQL Server). In MySQL, it's group_concat. This function takes a column of values and concatenates them into a single string. It's useful for aggregation queries.

Syntax:
STRING_AGG ( expression, separator ) [ WITHIN GROUP ( <order_by_clause> ) ]
- expression: The expression to be concatenated.
- separator: The separator to be used between concatenated values.
- WITHIN GROUP: Optional. Sorts the values that are concatenated.

What is...?
* WITHIN GROUP: The clause is used to specify an order for an ordered-set aggregate function. This is often used in conjunction with aggregation functions like STRING_AGG to specify the order in which the aggregated strings should appear.

Syntax: 
WITHIN GROUP ( ORDER BY expression [ ASC | DESC ] )

Create & insert the data

This is the data you can use to check your code with. You may create your own as well.

As always, I use Microsoft’s SSMS (SQL Server Management Studio), so it’s in Transact SQL.

-- Create the database
CREATE DATABASE tic_tac_toe_challenge;
USE tic_tac_toe_challenge;
--
--drop table horizontal_win_1
-- Create the horizontal_win_1 table
CREATE TABLE horizontal_win_1 (
    row_num INT PRIMARY KEY,
    A VARCHAR(5),
    B VARCHAR(5),
    C VARCHAR(5)
);
-- Insert data into horizontal_win_1
INSERT INTO horizontal_win_1 (row_num, A, B, C) VALUES
(1, 'O', 'X', NULL),
(2, 'X', 'X', NULL),
(3, 'O', 'O', NULL)
--
--drop table vertical_win_1
-- Create the vertical_win_1 table
CREATE TABLE vertical_win_1 (
    row_num INT PRIMARY KEY,
    A VARCHAR(5),
    B VARCHAR(5),
    C VARCHAR(5)
);
-- Insert data into vertical_win_1
INSERT INTO vertical_win_1 (row_num, A, B, C) VALUES
(1, 'X', 'O', NULL),
(2, 'X', 'O', 'O'),
(3, NULL, NULL, 'X')
--
--
--drop table vertical_win_2
-- Create the vertical_win_2 table
CREATE TABLE vertical_win_2 (
    row_num INT PRIMARY KEY,
    A VARCHAR(5),
    B VARCHAR(5),
    C VARCHAR(5)
);
-- Insert data into vertical_win_2
INSERT INTO vertical_win_2 (row_num, A, B, C) VALUES
(1, 'O', 'O', NULL),
(2, NULL, 'O', 'X'),
(3, 'X', NULL, 'X')
--
--drop table diagonal_win_1
-- Create the diagonal_win_1 table
CREATE TABLE diagonal_win_1 (
    row_num INT PRIMARY KEY,
    A VARCHAR(5),
    B VARCHAR(5),
    C VARCHAR(5)
);
-- Insert data into diagonol_win_1
INSERT INTO diagonol_win_1 (row_num, A, B, C) VALUES
(1, 'X', 'O', NULL),
(2, 'X', 'X', 'O'),
(3, 'O', NULL, NULL)
--
--drop table diagonal_win_2
-- Create the diagonal_win_2 table
CREATE TABLE diagonal_win_2 (
    row_num INT PRIMARY KEY,
    A VARCHAR(5),
    B VARCHAR(5),
    C VARCHAR(5)
);
-- Insert data into diagonal_win_2
INSERT INTO diagonal_win_2 (row_num, A, B, C) VALUES
(1, 'O', NULL, 'X'),
(2, NULL, 'X', NULL),
(3, NULL, NULL, 'O')
--
--drop table double_win_vertical_and_diagonal
-- Create the double_win_vertical_and_diagonal table
CREATE TABLE double_win_vertical_and_diagonal (
    row_num INT PRIMARY KEY,
    A VARCHAR(5),
    B VARCHAR(5),
    C VARCHAR(5)
);
-- Insert data into double_win_vertical_and_diagonal
INSERT INTO double_win_vertical_and_diagonal (row_num, A, B, C) VALUES
(1, 'X', 'X', 'O'),
(2, 'O', 'X', NULL),
(3, 'O', NULL, NULL)
;
----

SQL Solution #1

I suppose the thing about this solution is that it’s simple stupid. All you have to do is understand there are 3 options: horizontal wins, vertical wins, and diagonals. Then, there are 3 horizontal rows with 3 different options within each, and similarly same situation for the vertical ones. Finally, 2 diagonals, 3 scenarios each.

So once you write the solution for one horizontal solution, one vertical and one diagonal, you can copy paste the code and do minor edits to wrap it all up.

It’s doable for sharp cookies. Yet it would also allow managers to evaluate progress and planning, attention to details, thoroughness, cleanness of code (or ‘readability’, which could be a major factor in not making a silly mistake here), et cetera, for folks who wouldn’t necessarily finish it all up by the allotted time.

Subscribe now

--CTE for solution breakdown and comfort
WITH tic_tac_toe AS (
SELECT *
FROM TABLE_NAME -- change name here once for chosen game table
)
, winning_move_checks AS (
-- Horizontal wins
SELECT CONCAT('PLAY ', row_num, 'A') AS WinningMove
FROM tic_tac_toe
WHERE A IS NULL AND B = 'X' AND C = 'X'
UNION
SELECT CONCAT('PLAY ', row_num, 'B') AS WinningMove
FROM tic_tac_toe
WHERE A = 'X' AND B IS NULL AND C = 'X'
UNION
SELECT CONCAT('PLAY ', row_num, 'C') AS WinningMove
FROM tic_tac_toe
WHERE A = 'X' AND B = 'X' AND C IS NULL
---- Vertical wins
-- for A column
UNION
SELECT 'PLAY 1A' AS WinningMove
FROM tic_tac_toe
WHERE A IS NULL AND 
	(SELECT A FROM tic_tac_toe WHERE row_num = 2) = 'X' AND 
	(SELECT A FROM tic_tac_toe WHERE row_num = 3) = 'X'
UNION
SELECT 'PLAY 2A' AS WinningMove
FROM tic_tac_toe
WHERE A = 'X' AND 
	(SELECT A FROM tic_tac_toe WHERE row_num = 2) IS NULL AND 
	(SELECT A FROM tic_tac_toe WHERE row_num = 2) = 'X'
UNION
SELECT 'PLAY 3A' AS WinningMove
FROM tic_tac_toe
WHERE (SELECT A FROM tic_tac_toe WHERE row_num = 1) = 'X' AND 
	(SELECT A FROM tic_tac_toe WHERE row_num = 2) = 'X' AND 
	(SELECT A FROM tic_tac_toe WHERE row_num = 3) IS NULL
-- for B column
UNION
SELECT 'PLAY 1B' AS WinningMove
FROM tic_tac_toe
WHERE B IS NULL AND 
	(SELECT B FROM tic_tac_toe WHERE row_num = 2) = 'X' AND 
	(SELECT B FROM tic_tac_toe WHERE row_num = 3) = 'X'
UNION
SELECT 'PLAY 2B' AS WinningMove
FROM tic_tac_toe
WHERE B = 'X' AND 
	(SELECT B FROM tic_tac_toe WHERE row_num = 2) IS NULL AND 
	(SELECT B FROM tic_tac_toe WHERE row_num = 2) = 'X'
UNION
SELECT 'PLAY 3B' AS WinningMove
FROM tic_tac_toe
WHERE (SELECT B FROM tic_tac_toe WHERE row_num = 1) = 'X' AND 
	(SELECT B FROM tic_tac_toe WHERE row_num = 2) = 'X' AND 
	(SELECT B FROM tic_tac_toe WHERE row_num = 3) IS NULL
-- for C column
UNION
SELECT 'PLAY 1C' AS WinningMove
FROM tic_tac_toe
WHERE C IS NULL AND 
	(SELECT C FROM tic_tac_toe WHERE row_num = 2) = 'X' AND 
	(SELECT C FROM tic_tac_toe WHERE row_num = 3) = 'X'
UNION
SELECT 'PLAY 2C' AS WinningMove
FROM tic_tac_toe
WHERE C = 'X' AND 
	(SELECT C FROM tic_tac_toe WHERE row_num = 2) IS NULL AND 
	(SELECT C FROM tic_tac_toe WHERE row_num = 2) = 'X'
UNION
SELECT 'PLAY 3C' AS WinningMove
FROM tic_tac_toe
WHERE (SELECT C FROM tic_tac_toe WHERE row_num = 1) = 'X' AND 
	(SELECT C FROM tic_tac_toe WHERE row_num = 2) = 'X' AND 
	(SELECT C FROM tic_tac_toe WHERE row_num = 3) IS NULL
---- Diagonal wins
-- first diagonal
-- |[_]| ~ | ~ |
-- | ~ |[_]| ~ |
-- | ~ | ~ |[_]|
UNION
SELECT 'PLAY 1A' AS WinningMove
FROM tic_tac_toe
WHERE (SELECT A FROM tic_tac_toe WHERE row_num = 1) IS NULL AND 
	(SELECT B FROM tic_tac_toe WHERE row_num = 2) = 'X' AND 
	(SELECT C FROM tic_tac_toe WHERE row_num = 3) = 'X'
UNION
SELECT 'PLAY 2B' AS WinningMove
FROM tic_tac_toe
WHERE (SELECT A FROM tic_tac_toe WHERE row_num = 1) = 'X' AND 
	(SELECT B FROM tic_tac_toe WHERE row_num = 2) IS NULL AND 
	(SELECT C FROM tic_tac_toe WHERE row_num = 3) = 'X'
UNION
SELECT 'PLAY 3C' AS WinningMove
FROM tic_tac_toe
WHERE (SELECT A FROM tic_tac_toe WHERE row_num = 1) = 'X' AND 
	(SELECT B FROM tic_tac_toe WHERE row_num = 2) = 'X' AND 
	(SELECT C FROM tic_tac_toe WHERE row_num = 3) IS NULL
-- second diagonal
-- | ~ | ~ |[_]|
-- | ~ |[_]| ~ |
-- |[_]| ~ | ~ |
UNION
SELECT 'PLAY 1C' AS WinningMove
FROM tic_tac_toe
WHERE (SELECT C FROM tic_tac_toe WHERE row_num = 1) IS NULL AND 
	(SELECT B FROM tic_tac_toe WHERE row_num = 2) = 'X' AND
	(SELECT A FROM tic_tac_toe WHERE row_num = 3) = 'X'
UNION
SELECT 'PLAY 2B' AS WinningMove
FROM tic_tac_toe
WHERE (SELECT C FROM tic_tac_toe WHERE row_num = 1) = 'X' AND 
	(SELECT B FROM tic_tac_toe WHERE row_num = 2) IS NULL AND 
	(SELECT A FROM tic_tac_toe WHERE row_num = 3) = 'X'
UNION
SELECT 'PLAY 3A' AS WinningMove
FROM tic_tac_toe
WHERE (SELECT C FROM tic_tac_toe WHERE row_num = 1) = 'X' AND 
	(SELECT B FROM tic_tac_toe WHERE row_num = 2) = 'X' AND
	(SELECT A FROM tic_tac_toe WHERE row_num = 3) IS NULL
)
SELECT *
FROM winning_move_checks
;

Share

SQL Solution #2

--CTE for solution breakdown and comfort
WITH tic_tac_toe AS (
SELECT *
FROM TableName -- change name here once for chosen game table
)
-- For horizontal wins
, RowStrings AS (
    SELECT row_num,
           CONCAT(ISNULL(A, 'N'), ISNULL(B, 'N'), ISNULL(C, 'N')) AS row_string
    FROM tic_tac_toe
), horizontal_check AS (
SELECT 
    CASE 
        WHEN CHARINDEX('XXN', row_string) > 0 
         THEN CONCAT('PLAY ', row_num, 'C (horizontal)')
        WHEN CHARINDEX('XNX', row_string) > 0 
         THEN CONCAT('PLAY ', row_num, 'B (horizontal)')
        WHEN CHARINDEX('NXX', row_string) > 0 
         THEN CONCAT('PLAY ', row_num, 'A (horizontal)')
    END AS WinningMove
FROM RowStrings
WHERE row_string LIKE '%XXN%' OR 
	row_string LIKE '%XNX%' OR 
	row_string LIKE '%NXX%'
)
-- For vertical wins
, ColStrings AS (
    SELECT col_name
           ,STRING_AGG(CASE WHEN col IS NULL THEN 'N' ELSE col END, '') 
		WITHIN GROUP (ORDER BY row_num) AS col_string
    FROM (
        SELECT row_num, A as col, 'A' as col_name FROM tic_tac_toe
        UNION ALL
        SELECT row_num, B, 'B' FROM tic_tac_toe
        UNION ALL
        SELECT row_num, C, 'C' FROM tic_tac_toe
    ) AS subq
    GROUP BY col_name
)
, vertical_check AS (
SELECT CASE 
        WHEN CHARINDEX('XXN', col_string) > 0 
         THEN CONCAT('PLAY 3', col_name, ' (vertical)')
        WHEN CHARINDEX('XNX', col_string) > 0 
         THEN CONCAT('PLAY 2', col_name, ' (vertical)')
        WHEN CHARINDEX('NXX', col_string) > 0 
         THEN CONCAT('PLAY 1', col_name, ' (vertical)')
    END AS WinningMove
FROM ColStrings
WHERE col_string LIKE '%XXN%' OR 
	col_string LIKE '%XNX%' OR 
	col_string LIKE '%NXX%'
)
-- For diagonal wins
, DiagStrings AS (
-- MAIN diagonal
-- |[_]| ~ | ~ |
-- | ~ |[_]| ~ |
-- | ~ | ~ |[_]|
-- counter diagonal
-- | ~ | ~ |[_]|
-- | ~ |[_]| ~ |
-- |[_]| ~ | ~ |
SELECT 'Main' AS diag_type
, STRING_AGG(CASE WHEN col IS NULL THEN 'N' ELSE col END, '') 
	WITHIN GROUP (ORDER BY row_num) AS diag_string
    FROM (
        SELECT row_num, A as col FROM tic_tac_toe WHERE row_num = 1
        UNION ALL
        SELECT row_num, B FROM tic_tac_toe WHERE row_num = 2
        UNION ALL
        SELECT row_num, C FROM tic_tac_toe WHERE row_num = 3
    ) AS subq
    UNION ALL
SELECT 'Counter' AS diag_type
, STRING_AGG(CASE WHEN col IS NULL THEN 'N' ELSE col END, '') 
	WITHIN GROUP (ORDER BY row_num) AS diag_string
    FROM (
        SELECT row_num, C as col FROM tic_tac_toe WHERE row_num = 1
        UNION ALL
        SELECT row_num, B FROM tic_tac_toe WHERE row_num = 2
        UNION ALL
        SELECT row_num, A FROM tic_tac_toe WHERE row_num = 3
    ) AS subquery
)
, diagonal_check AS (
SELECT 
    CASE 
        WHEN diag_type = 'Main' AND CHARINDEX('XXN', diag_string) > 0 
         THEN 'PLAY 3C (main diagonal)'
        WHEN diag_type = 'Main' AND CHARINDEX('XNX', diag_string) > 0 
         THEN 'PLAY 2B (main diagonal)'
        WHEN diag_type = 'Main' AND CHARINDEX('NXX', diag_string) > 0 
         THEN 'PLAY 1A (main diagonal)'
        WHEN diag_type = 'Counter' AND CHARINDEX('XXN', diag_string) > 0 
         THEN 'PLAY 3A (counter diagonal)'
        WHEN diag_type = 'Counter' AND CHARINDEX('XNX', diag_string) > 0 
         THEN 'PLAY 2B (counter diagonal)'
        WHEN diag_type = 'Counter' AND CHARINDEX('NXX', diag_string) > 0 
         THEN 'PLAY 1C (counter diagonal)'
    END AS WinningMove
FROM DiagStrings
WHERE diag_string LIKE '%XXN%' OR 
	diag_string LIKE '%XNX%' OR 
	diag_string LIKE '%NXX%'
)
, winning_moves_checks AS (
SELECT *
FROM horizontal_check
union
SELECT *
FROM vertical_check
union
SELECT *
FROM diagonal_check
)
SELECT DISTINCT *
FROM winning_moves_checks
;

Share

Share novsto

Subscribe now

That’s it :) I hope you enjoyed it. Feel free to leave a comment if you’ve found an issue or got a suggestion. Alternatively, contact me via LinkedIn if you wish.

Practice #1 and #2 are quite easy. This practice, #3, is more advanced. I suppose this might be considered like an assignment a company may give with 1-2 days to return, perhaps with the last question as a bonus question.

• UPDATE; SET; CASE WHEN; CTE; UNION ALL; DATE FUNCTIONS; COUNT(); GROUP BY; HAVING; JOIN; FULL OUTER JOIN; COALESCE; SUM; WINDOW FUNCTIONS; WHERE; ORDER BY; CAST; FLOOR; NULLIF; TEMPORARY TABLE; ROUND; And more.

You can download the datasets below.

Share novsto

Subscribe now

Share

The Data

Download the three datasets (CSV):

1. Tibia Kill Statistics.

2. Tibia Game World Servers.

3. Tibia Deprecated Servers. 

You may copy the table creation and data loading from Practice #1.

Practice #3:

You’ll be using all datasets. Decide which to use on your own.

The Questions

Question #1:  The deprecated servers (old servers dataset) are missing server location. Analyze the “worlds” dataset to figure out which (old) server was located where. The server world name has a structure matching with server location. 

Then, update the “oldservers” dataset accordingly.

• If you need, see hints at “The Answers” section below. Scroll slowly if you don’t want to be exposed to both hints/the answer at once.

Question #2: Create a single visualization which presents the journey of expansion of CipSoft's Tibia to new geographical areas via the server locations throughout the game's existence.

Although there is no one "correct" answer, ensure that with your query/queries, the visualization is clear about how many servers Tibia had at least per year throughout its existence and the geographical locations.

Note: I propose two versions for this question. One, visualizing with Tableau, and the other, visualizing using Excel. The required output for fast visualization with Excel essentially warrants a somewhat more thoughtful query in my opinion. So consider tackling both versions.

Question #3: Find out how many (regular) creatures and how many (unique monsters) bosses there are, given that all "races" are either Creatures, Bosses or Players.

Note that you can distinguish between Creatures and Bosses based on the fact Bosses' names start with Capitalized letters while the rest do not.

Desired output:

Question 4#: Find the 15 most lethal races (from most to least lethal), by their rank, displaying their Lethality Rank, Race name, and Lethality Rate (players killed per race killed). Adjust the Lethality Rate as shown in the desired output image.

Desired output:

Question #5: Produce the output as shown below. It is a table which displays the Amount of, and overall Percentage in the population, the lethality rate of, the different races once classified into groups. 

• Arrange the data into six different Lethality Group categories as shown.

• Pay attention that your output produces LethalityGroup E. 

• Perform the following data cleaning steps as well, and account for it in your calculations:

  • Some bosses evolve in stages during the boss fight and thus, require to merge the different names they appear under into one. Use the following list to handle it:
    
    - 'Urmahlullu the Immaculate', 'Wisdom of Urmahlullu', 'Urmahlullu the Tamed', 'Wildness of Urmahlullu' = 'Urmahlullu the Weakened'
    
    -  'Dragonking Zyrtarch' = 'Soul of Dragonking Zyrtarch'
    
    - 'Channeling Earl Osam' = 'Earl Osam'
    
    -  'Charged Anomaly' = 'Anomaly'
    
    - 'glooth powered minotaurs' = 'Depowered Minotaur'
    
    - 'The Fire Empowered Duke' = 'The Duke Of The Depths'
    

  • Remove races which are game mechanisms, and therefore, cannot be killed (e.g. elements such as fire or magic pillars). You can spot them by the fact they cannot, and therefore never are, killed.

• Consider a minimum threshold of 30 "encounters" in your evaluation (encounter being one of the parties is killed- players or creatures)

Desired output:

Question #6: Produce the same final output of the previous question (question #5), but this time, in the formation as shown at the desired output.

• General note:
  SQL does not have Transpose. If you want to convert data for more than 2 columns, it's usually easiest to export the output to e.g. Excel or use Python.
  Yet consider the following challenge here and try it out. I found it to be an interesting thought-exercise and a way to instill stronger understanding of some concepts.
  That noted, I’m sure in the future I’ll find out there was some easier/smarter way to do this :) A way I simply don’t know/haven’t thought of at this very moment.
  But if you know of a better way, please leave a comment!

  Leave a comment

Desired output:

or if you prefer to see how it looks in Excel:

(Bonus?) Question 7#: Examine players' deaths at different servers (PvP Type & Location). Provide one or two interesting insight(s).

• No specific desired output.

• No specific solution code.

The Answers

Hint #1 for Question #1:

• Hint 1: The pattern is most clear for the South America ones, then North America ones, and least for the European servers.

Hint #2 for Question #1:

• Focus on the name ending.

Answer #1:  The deprecated servers (old servers dataset) are missing server location. Analyze the “worlds” dataset to figure out which (old) server was located where. The server world name has a structure matching with server location. 

~bra = South America (servers located in Brazil)
~era = North America (servers located in North America)
~a = Europe (servers located in Europe; names ending with A which don't end with BRA nor ERA)

UPDATE oldservers	
SET located = 	
    CASE	
        WHEN servName LIKE '%bra' THEN 'South America'	
        WHEN servName LIKE '%era' THEN 'North America'	
        ELSE 'Europe'	
    END

Answer #2: Create a single visualization which presents the journey of expansion of CipSoft's Tibia to new geographical areas via the server locations throughout the game's existence.

Tableau version - desired output example:

Solution code below:

WITH old_and_current AS (		
  select servName		
  , located		
  , YEAR(onlineSince) AS startYear		
  , YEAR(offlineSince) AS endYear		
  from oldservers		
    union all		
  select worldName		
  , servLocation		
  , YEAR(serverCETCreatedDate) AS startYear		
  , CASE WHEN serverCETCreatedDate is not null THEN '0000' END AS endYear		
  from worlds		
  )		
  , start_year AS (		
  SELECT 		
  startYear		
  , located		
  , COUNT(servName) AS sYcounter		
  FROM old_and_current		
  GROUP BY 		
  startYear		
  , located		
  )		
  , end_year AS (		
  SELECT 		
  endYear		
  , located		
  , COUNT(servName) AS eYcounter		
  FROM old_and_current		
  GROUP BY 		
  endYear		
  , located		
  )		
 , yearly_status AS (		
  SELECT 		
  startYear AS "Year"		
  , start_year.located AS located		
  , COALESCE((sYcounter - eYcounter), sYcounter) AS NumStatus		
  FROM start_year		
  FULL OUTER JOIN end_year		
  ON start_year.startYear = end_year.endYear 		
  AND start_year.located = end_year.located		
  )		
  SELECT 		
  "Year"		
  , located		
  , SUM(NumStatus) OVER (PARTITION BY located ORDER BY "Year") AS Total		
  FROM yearly_status		
  WHERE "Year" IS NOT NULL		
  ORDER BY 2, 1

Excel version - desired output example:

Solution code below:

WITH old_and_current AS (
  SELECT servName			
  , located			
  , YEAR(onlineSince) AS startYear			
  , COALESCE(YEAR(offlineSince), 0) AS endYear			
  FROM oldservers			
  UNION ALL			
  SELECT worldName			
  , servLocation			
  , YEAR(serverCETCreatedDate) AS startYear			
  , 0 AS endYear			
  FROM worlds			
),			
addition AS (			
  SELECT startYear			
  , SUM(CASE WHEN located = 'Europe' THEN 1 ELSE 0 END) AS EuropePlus			
  , SUM(CASE WHEN located = 'North America' THEN 1 ELSE 0 END) AS NorthAmericaPlus			
  , SUM(CASE WHEN located = 'South America' THEN 1 ELSE 0 END) AS SouthAmericaPlus			
  FROM old_and_current			
  GROUP BY startYear			
  ),			
subtraction AS (			
  SELECT endYear			
  , SUM(CASE WHEN located = 'Europe' THEN 1 ELSE 0 END) AS EuropeMinus			
  , SUM(CASE WHEN located = 'North America' THEN 1 ELSE 0 END) AS NorthAmericaMinus			
  , SUM(CASE WHEN located = 'South America' THEN 1 ELSE 0 END) AS SouthAmericaMinus			
  FROM old_and_current			
  WHERE endYear != 0			
  GROUP BY endYear			
  ), 			
trendPerYear AS ( 			
  SELECT 			
  startYear AS "Year"			
  , COALESCE((EuropePlus - EuropeMinus), EuropePlus) AS EuropeTrend			
  , COALESCE((NorthAmericaPlus - NorthAmericaMinus), NorthAmericaPlus) AS NorthAmericaTrend			
  , COALESCE((SouthAmericaPlus - SouthAmericaMinus), SouthAmericaPlus) AS SouthAmericaTrend			
  FROM addition			
  FULL OUTER JOIN subtraction			
  ON addition.startYear = subtraction.endYear			
 ),			
statusPerYear AS (			
 SELECT 			
 "Year"			
 , SUM(EuropeTrend) OVER (ORDER BY "Year") AS Europe			
 , SUM(NorthAmericaTrend) OVER (ORDER BY "Year") AS NorthAmerica			
 , SUM(SouthAmericaTrend) OVER (ORDER BY "Year") AS SouthAmerica			
 FROM trendPerYear			
 )					
 SELECT "Year"		
 , Europe			
 , NorthAmerica			
 , SouthAmerica
 , (Europe + NorthAmerica + SouthAmerica) AS Total				
 FROM statusPerYear

Answer 3#: Find out how many (regular) creatures and how many (unique monsters) bosses there are, given that all "races" are either Creatures, Bosses or Players.

Solution code below:

SELECT 		
 COUNT(CASE WHEN ASCII(LEFT(race, 1)) BETWEEN 65 AND 90 THEN 1 END ) AS 'Boss'		
 , COUNT(CASE WHEN ASCII(LEFT(race, 1)) NOT BETWEEN 65 AND 90 THEN 1 END ) AS 'Creature'		
FROM (SELECT distinct race 		
      FROM killstats		
	  WHERE race != 'players'	
	  ) source

Answer: 4#: Find the 15 most lethal races (from most to least lethal), by their rank, displaying their Lethality Rank, Race name, and Lethality Rate (players killed per race killed). Adjust the Lethality Rate as shown in the desired output image.

Solution code below:

WITH LethalityTable AS (		
    SELECT 		
    race		
    , COALESCE(		
        CAST(		
           FLOOR(		
              CAST(SUM(lastWeekKilledPlayers) AS DECIMAL(18, 6)) 		
	       / 	
	       NULLIF	
	         (CAST(SUM(lastWeekKilledbyPlayers) AS DECIMAL(18, 6))	
	          , 0) --nullIF 	
	        * 1000 --for FLOOR	
                ) / 1000 --for FLOOR		
            AS DECIMAL(18, 3))		
	, 0) -- coalesce	
	AS LethalityRate	
    FROM killstats		
    GROUP BY race		
)		
SELECT 		
    TOP 16 		
    DENSE_RANK() OVER (ORDER BY LethalityRate DESC) AS LethalityRank		
    , Race		
    , LethalityRate		
FROM LethalityTable		
ORDER BY LethalityRate DESC

Answer #5: Produce the output as shown below. It is a table which displays the Amount of, and overall Percentage in the population, the lethality rate of, the different races once classified into groups. i.e. according to the described details.

Solution code below:

DROP TABLE IF EXISTS #all_categories;			
CREATE TABLE #all_categories			
(			
    LethalityGroup VARCHAR(50)			
);			
INSERT INTO #all_categories(LethalityGroup)			
VALUES ('(A) 0-0.99'), ('(B) 1-9.99'), ('(C) 10-49.99'), ('(D) 50-89.99'), ('(E) 90-99.99'), ('(F) 100+');			
			
WITH AggregatedTable AS (--Grouped for aggregation, changed integer into decimal for rates			
    SELECT 			
        serverName			
        , race			
        , CAST(SUM(lastWeekKilledPlayers) AS DECIMAL(13,3)) AS sumKilledPlayers			
        , CAST(SUM(lastWeekKilledbyPlayers) AS DECIMAL(13,3)) AS sumPlayersKilled			
    FROM killstats			
    GROUP BY 			
        serverName			
        , race			
)			
, StagedBossesMerger AS (--sumPlayersKilled equal zero reveals list of creatures which 			
    --cannot be killed at all or that got different forms before they can be killed in their final form.			
   --the CASE WHEN merged the different forms into the Killable-form.			
   --the HAVING part removed the UnKillable Creatures.			
   --And yes, to figure it out, there was a need to manual check on wiki-fandom website.			
  SELECT 			
    CASE			
        WHEN race IN ('Urmahlullu the Immaculate', 'Wisdom of Urmahlullu', 'Urmahlullu the Tamed', 'Wildness of Urmahlullu') THEN 'Urmahlullu the Weakened'			
        WHEN race IN ('Dragonking Zyrtarch') THEN 'Soul of Dragonking Zyrtarch'			
        WHEN race IN ('Channeling Earl Osam') THEN 'Earl Osam'			
        WHEN race IN ('Charged Anomaly') THEN 'Anomaly'			
        WHEN race IN ('glooth powered minotaurs') THEN 'Depowered Minotaur'			
        WHEN race IN ('The Fire Empowered Duke') THEN 'The Duke Of The Depths'			
        ELSE race			
    END AS race			
    , SUM(sumKilledPlayers) AS sumKilledPlayers			
	, SUM(sumPlayersKilled) AS sumPlayersKilled		
FROM AggregatedTable			
GROUP BY			
    CASE			
        WHEN race IN ('Urmahlullu the Immaculate', 'Wisdom of Urmahlullu', 'Urmahlullu the Tamed', 'Wildness of Urmahlullu') THEN 'Urmahlullu the Weakened'			
        WHEN race IN ('Dragonking Zyrtarch') THEN 'Soul of Dragonking Zyrtarch'			
        WHEN race IN ('Channeling Earl Osam') THEN 'Earl Osam'			
        WHEN race IN ('Charged Anomaly') THEN 'Anomaly'			
        WHEN race IN ('glooth powered minotaurs') THEN 'Depowered Minotaur'			
        WHEN race IN ('The Fire Empowered Duke') THEN 'The Duke Of The Depths'			
        ELSE race			
    END 			
	HAVING SUM(sumPlayersKilled) > 0		
	)		
	, LethalityTable AS (		
	SELECT race		
	, ROUND((sumKilledPlayers/sumPlayersKilled)*100,2) AS Lethality		
    FROM StagedBossesMerger			
    WHERE sumKilledPlayers+sumPlayersKilled >= 30 ---threshold			
    )			
	, LethalityTable2 AS (		
    SELECT			
      cate.LethalityGroup			
	  , ISNULL(COUNT(l.race), 0) AS Amount		
    FROM LethalityTable l			
    RIGHT JOIN #all_categories cate			
    ON cate.LethalityGroup = 			
      CASE WHEN l.Lethality BETWEEN 0 AND 0.99 THEN '(A) 0-0.99'			
	    WHEN l.Lethality BETWEEN 1 AND 9.99 THEN '(B) 1-9.99'		
	    WHEN l.Lethality BETWEEN 10 AND 49.99 THEN '(C) 10-49.99'		
	    WHEN l.Lethality BETWEEN 50 AND 89.99 THEN '(D) 50-89.99'		
	    WHEN l.Lethality BETWEEN 90 AND 99.99 THEN '(E) 90-99.99'		
	    WHEN l.Lethality >= 100 THEN '(F) 100+'		
      END 			
    GROUP BY			
      cate.LethalityGroup			
	  ),		
	  LethalityTable3 AS (		
	  SELECT LethalityGroup		
	       , Amount		
	   , CAST(Amount AS DECIMAL(10,4))		
	     /		
	 (SELECT CAST(SUM(Amount) AS DECIMAL(10,4))		
	      FROM LethalityTable2)*100		
	  AS "Percent"		
       FROM LethalityTable2			
	   )		
SELECT LethalityGroup			
	   , Amount		
	   , ROUND("Percent",2) AS "Percent"		
FROM LethalityTable3

Answer #6: Produce the same final output of the previous question (question #5), but this time, in the formation as shown at the desired output.

Solution code below:

WITH AggregatedTable AS ( --Grouped for aggregation and changed integer into decimal for rates		
    SELECT 		
        serverName		
        , race		
        , CAST(SUM(lastWeekKilledPlayers) AS DECIMAL(13,3)) AS sumKilledPlayers		
        , CAST(SUM(lastWeekKilledbyPlayers) AS DECIMAL(13,3)) AS sumPlayersKilled		
    FROM killstats		
    GROUP BY 		
        serverName		
        , race		
)		
, StagedBossesMerger AS (--sumPlayersKilled equal zero reveals list of creatures which 		
    --cannot be killed at all or that got different forms before they can be killed in their final form.		
   --the CASE WHEN merged the different forms into the Killable-form.		
   --the HAVING part removed the UnKillable Creatures.		
   --And yes, to figure it out, there was a need to manual check on wiki-fandom website.		
  SELECT 		
    CASE		
        WHEN race IN ('Urmahlullu the Immaculate', 'Wisdom of Urmahlullu', 'Urmahlullu the Tamed', 'Wildness of Urmahlullu') THEN 'Urmahlullu the Weakened'		
        WHEN race IN ('Dragonking Zyrtarch') THEN 'Soul of Dragonking Zyrtarch'		
        WHEN race IN ('Channeling Earl Osam') THEN 'Earl Osam'		
        WHEN race IN ('Charged Anomaly') THEN 'Anomaly'		
        WHEN race IN ('glooth powered minotaurs') THEN 'Depowered Minotaur'		
        WHEN race IN ('The Fire Empowered Duke') THEN 'The Duke Of The Depths'		
        ELSE race		
    END AS race		
    , SUM(sumKilledPlayers) AS sumKilledPlayers		
	, SUM(sumPlayersKilled) AS sumPlayersKilled	
FROM AggregatedTable		
GROUP BY		
    CASE		
        WHEN race IN ('Urmahlullu the Immaculate', 'Wisdom of Urmahlullu', 'Urmahlullu the Tamed', 'Wildness of Urmahlullu') THEN 'Urmahlullu the Weakened'		
        WHEN race IN ('Dragonking Zyrtarch') THEN 'Soul of Dragonking Zyrtarch'		
        WHEN race IN ('Channeling Earl Osam') THEN 'Earl Osam'		
        WHEN race IN ('Charged Anomaly') THEN 'Anomaly'		
        WHEN race IN ('glooth powered minotaurs') THEN 'Depowered Minotaur'		
        WHEN race IN ('The Fire Empowered Duke') THEN 'The Duke Of The Depths'		
        ELSE race		
    END 		
	HAVING SUM(sumPlayersKilled) > 0	
	)	
	, LethalityTable AS (	
	SELECT race	
	, ROUND((sumKilledPlayers/sumPlayersKilled)*100,2) AS Lethality	
    FROM StagedBossesMerger		
    WHERE sumKilledPlayers+sumPlayersKilled >= 30 ---threshold		
    ),		
	LethalityAmount AS (	
	SELECT	
    COUNT(CASE WHEN Lethality BETWEEN 0 AND 0.99 THEN 1 END) 
    AS '(A) 0-0.99'		
	, COUNT(CASE WHEN Lethality BETWEEN 1 AND 9.99 THEN 1 END) 
    AS '(B) 1-9.99'	
	, COUNT(CASE WHEN Lethality BETWEEN 10 AND 49.99 THEN 1 END) 
    AS '(C) 10-49.99'	
	, COUNT(CASE WHEN Lethality BETWEEN 50 AND 89.99 THEN 1 END) 
    AS '(D) 50-89.99'	
	, COUNT(CASE WHEN Lethality BETWEEN 90 AND 99.99 THEN 1 END) 
    AS '(E) 90-99.99'	
	, COUNT(CASE WHEN Lethality >= 100 THEN 1 END) 
    AS '(F) 100+'	
	FROM LethalityTable	
	),	
	LethalityPercent AS (	
	SELECT 	
	 (SELECT CAST(COUNT(CASE WHEN Lethality BETWEEN 0 AND 0.99 THEN 1 END) AS DECIMAL(8,2)) FROM LethalityTable)	
	  /	
	 (SELECT CAST(COUNT(race) AS DECIMAL(8,2)) FROM LethalityTable)	
	  AS '(A) 0-0.99'	
	 ,	
	 (SELECT CAST(COUNT(CASE WHEN Lethality BETWEEN 1 AND 9.99 THEN 1 END) AS DECIMAL(8,2)) FROM LethalityTable)	
	  /	
	 (SELECT CAST(COUNT(race) AS DECIMAL(8,2)) FROM LethalityTable)	
	  AS '(B) 1-9.99'	
	 ,	
	 (SELECT CAST(COUNT(CASE WHEN Lethality BETWEEN 10 AND 49.99 THEN 1 END) AS DECIMAL(8,2)) FROM LethalityTable)	
	  /	
	 (SELECT CAST(COUNT(race) AS DECIMAL(8,2)) FROM LethalityTable)	
	  AS '(C) 10-49.99'	
	 ,	
	 (SELECT CAST(COUNT(CASE WHEN Lethality BETWEEN 50 AND 89.99 THEN 1 END) AS DECIMAL(8,2)) FROM LethalityTable)	
	  /	
	 (SELECT CAST(COUNT(race) AS DECIMAL(8,2)) FROM LethalityTable)	
	  AS '(D) 50-89.99'	
	  ,	
	 (SELECT CAST(COUNT(CASE WHEN Lethality BETWEEN 90 AND 99.99 THEN 1 END) AS DECIMAL(8,2)) FROM LethalityTable)	
	  /	
	 (SELECT CAST(COUNT(race) AS DECIMAL(8,2)) FROM LethalityTable)	
	  AS '(E) 90-99.99'	
	 ,	
	 (SELECT CAST(COUNT(CASE WHEN Lethality >= 100 THEN 1 END) AS DECIMAL(8,2)) FROM LethalityTable)	
	  /	
	 (SELECT CAST(COUNT(race) AS DECIMAL(8,2)) FROM LethalityTable)	
	  AS '(F) 100+'	
 )		
SELECT *		
FROM LethalityAmount		
UNION ALL		
SELECT *		
FROM LethalityPercent

(Bonus?) Answer 7#: Examine players' deaths at different servers (PvP Type & Location). Provide one or two interesting insight(s).

The insight I chose to show was that players on South American servers die to PvE (Player versus the Environment) significantly more than players on North American & European servers. This can be seen at Open PvP, Retro Open PvP and Retro Hardcore PvP. But I chose to illustrate this with the Open PvP servers:

I wonder if the reason for it is a bad internet infrastructure.

Anyhow, below is most of the SQL code used for the evaluation:

• Question #E

• And \/

WITH step1 AS (	
SELECT servLocation	
, pvpType	
, CASE WHEN (race = 'players') THEN 'PvP' --P for Players	
  ELSE 'PvE' --E for Environment 	
  END AS DeathType	
, SUM(lastWeekKilledPlayers) AS Total_Players_Killed_TYPE	
, CONCAT(	
   CAST(	
    FORMAT(	
	 ROUND(
          (	
          CAST(SUM(lastWeekKilledPlayers) AS DECIMAL(18,2))	
          / 	
          CAST(SUM(SUM(lastWeekKilledPlayers)) OVER (PARTITION BY pvpType, servLocation) AS DECIMAL(18,2))	
           )*100 --multiply by 100 for Percent	
     , 2) --round to 2 digits past period	
    , 'F' --to Format the digit in a way which eliminates the 0 trailing after the 2 rounded digits  	
   ) AS VARCHAR --turn to String for CONCAT to work 	
   ), '%' --added % sign with CONCAT to signal percent   	
   ) AS Percent_Players_Killed_TYPE	
FROM killstats	
JOIN worlds	
ON killstats.serverName = worlds.worldName	
GROUP BY servLocation	
  , pvpType	
  , CASE WHEN (race = 'players') THEN 'PvP'	
  ELSE 'PvE' END	
)	
SELECT pvpType	
      , servLocation	
	  , DeathType
	  , Percent_Players_Killed_TYPE
FROM step1	
ORDER BY pvpType DESC

Share novsto

Share

That’s it for the Tibia SQL Data Analysis Practice. I hope you enjoyed the questions and this practice.

The data gives ways to do more interesting analysis, I’m sure of it. Feel free to drop suggestions at the comments or contact me directly via e-mail or LinkedIn.

LinkedIn

Subscribe now

Leave a comment

You can download the datasets below.

Share novsto

Share

• 1997: The start

• 2002: Expansion to North America

• until 2005: Experiencing meteoric growth

• until 2013: Slower steady growth

• 2014: The big shrink

• 2016: Introduction of South American servers; slow recovery starts

• 2018: South America grows; Europe and North America continues to shrink

• 2021: South America outgrows Europe and North America

• 2023: All times high

Subscribe now

The Data

Download the three datasets (CSV):

1. Tibia Kill Statistics.

2. Tibia Game World Servers.

3. Tibia Deprecated Servers. 

You may copy the table creation and data loading from Practice #1.

Practice #2: Second SQL warm-up

For this warm up, use the dataset: Tibia Game Worlds Server

The Questions

(the answers are provided at the bottom, section called “The Answers”)

Question #F: What's the oldest server? Show: worldname, date, servLocation, pvpType.

Desired output:

Question #G: Which are the three newest servers? Show: worldname, date, location, pvpType.

Desired output:

Question #H: What is the newest and oldest server for each server location? Show: worldname, location, date. Display from the oldest to the newest.

Desired output:

Question #I: What's the newest and oldest server for each server location and PvP type? Show: server location ASC, PvO type DESC, date ASC, worldname.

Desired output:

The Answers

Answer #F: What's the oldest server? Show: worldname, date, servLocation, pvpType.

Solution code below:

SELECT TOP 1 worldName
  , serverCETCreatedDate
  , servLocation
  , pvpType
FROM worlds
ORDER BY 2 ASC

Answer #G: Which are the three newest servers? Show: worldname, date, location, pvpType.

Solution code below:

SELECT TOP 3 worldName
  , serverCETCreatedDate
  , servLocation
  , pvpType
FROM worlds
ORDER BY 2 DESC

Answer #H: What is the newest and oldest server for each server location? Show: worldname, location, date. Display from the oldest to the newest.

Solution code below:

WITH dateAndLocation AS (
SELECT MIN(serverCETCreatedDate) AS earliest	
       , servLocation	
       , MAX(serverCETCreatedDate) AS latest	
FROM worlds	
GROUP BY servLocation	
)	
SELECT worldname
       , worlds.servLocation
       , serverCETCreatedDate	
FROM worlds	
JOIN dateAndLocation	
ON worlds.servLocation = dateAndLocation.servLocation 	
       AND (
       (worlds.serverCETCreatedDate = dateAndLocation.earliest) 
       OR (worlds.serverCETCreatedDate = dateAndLocation.latest)
)	
ORDER BY 3	

Answer #I: What's the newest and oldest server for each server location and PvP type? Show: server location ASC, PvO type DESC, date ASC, worldname.

Solution code below:

WITH dateAndLocation AS (
SELECT MIN(serverCETCreatedDate) AS earliest
  , pvpType
  , servLocation
  , MAX(serverCETCreatedDate) AS latest
FROM worlds
GROUP BY servlocation, pvpType
)
SELECT worlds.servLocation
  , worlds.pvpType
  , serverCETCreatedDate
  , worldName
FROM worlds
JOIN dateAndLocation
ON worlds.pvpType = dateAndLocation.pvpType
  AND worlds.servLocation = dateAndLocation.servLocation
  AND (
  (worlds.serverCETCreatedDate = dateAndLocation.earliest) 
  OR (worlds.serverCETCreatedDate = dateAndLocation.latest)
)
ORDER BY 1, 2 DESC, 3

Share

Subscribe now

You can download the datasets below.

Share novsto

Share

Tibia1 is a massive multiplayer online role-playing game (MMORPG) that exists since 19972! (and also name of a bone3 =) ).

• 1997: The start

• 2002: Expansion to North America

• until 2005: Experiencing meteoric growth

• until 2013: Slower steady growth

• 2014: The big shrink

• 2016: Introduction of South American servers; slow recovery starts

• 2018: South America grows; Europe and North America continues to shrink

• 2021: South America outgrows Europe and North America

• 2023: All times high

Subscribe now

Get the data & load it

Download the three datasets (CSV):

1. Tibia Kill Statistics.

2. Tibia Game World Servers.

3. Tibia Deprecated Servers. 

I used Microsoft's SQL Server Management Studio,4 so you may need to convert the code from Transact SQL to whatever version you use. I suppose ChatGPT could do that fairly fast and easily.

Create the tables, load the data

--DROP TABLE "worlds"
IF OBJECT_ID('worlds', 'U') IS NOT NULL
DROP TABLE worlds
------
------
CREATE TABLE worlds (
   worldName VARCHAR(10) NOT NULL,
   PRIMARY KEY (worldName),
   onlineRecord INT,
   onlineRecordDate DATE, --- in CET timezone
   servLocation VARCHAR(15), --- "serv" as in short of "server"
   pvpType VARCHAR(25), ---PvP = Player versus Player
   serverCETCreatedDate DATE, ---CET = Central European Time; and
 ---Day set to 1st (DD/MM/YYYY) as default due to data inavailability
   additionalInformation VARCHAR(250)
);
-----
-----
---LOAD CSV Tibia Game World Servers to load worlds table (of table)
BULK INSERT worlds
FROM 'C:\Users\...\Tibia Game World Servers.csv'
WITH (
FORMAT='CSV',
FIRSTROW=2
);
------
------
------
--DROP TABLE "killstats"
IF OBJECT_ID('killstats', 'U') IS NOT NULL
DROP TABLE killstats
------
------
CREATE TABLE killstats (
 serverName VARCHAR(10) NOT NULL,
  -- PRIMARY KEY (serverName),
 race VARCHAR(30) NOT NULL,
 lastDayKilledPlayers INT,
 lastDayKilledbyPlayers INT,
 lastWeekKilledPlayers INT,
 lastWeekKilledbyPlayers INT,
 WeekNumber INT
);
-----
-----
---LOAD CSV Tibia Kill Statistics to load killstats table (of table)
BULK INSERT killstats
FROM 'C:\Users\...\Tibia Kill Statistics.csv'
WITH (
FORMAT='CSV',
FIRSTROW=2
);
------
------
------
--DROP TABLE "oldservers"
IF OBJECT_ID('oldservers', 'U') IS NOT NULL
DROP TABLE oldservers
------
------
CREATE TABLE oldservers (
 servName VARCHAR(10) NOT NULL,
 located VARCHAR(15),
 servType VARCHAR(20), ---PvP = Player versus Player
 onlineSince DATE,
 offlineSince DATE
);
-----
-----
---LOAD CSV Tibia Deprecated Servers to load oldservers table (of table)
BULK INSERT oldservers
FROM 'C:\Users\...\Tibia Deprecated Servers.csv'
WITH (
FORMAT='CSV',
FIRSTROW=2
);
------
------
------

Practice #1: First SQL warm-up

For this warm up, use the dataset: Tibia Game Worlds Server

The Questions

(the answers are provided at the bottom, section called “The Answers”)

Question #A: How many game world servers are there currently in Tibia? 

Desired output:

  Number_of_online_worlds

  88

Question #B: How many unique PvP types of game world servers are there currently in Tibia?

Desired output:

  Pvp_Types_Number

  5

Question #C: How many current worlds are there per PvP Type?

Desired output:

Question #D: How many worlds are there currently in Tibia per server location?

Desired output:

Question #E: How many current online worlds are there per server location and PvP type?
Please display the results from the most to the least amount of servers.

Share

The Answers

Answer #A: How many game world servers are there currently in Tibia? 

Solution code below:

SELECT count(worldname) AS Number_of_online_worlds
FROM worlds

Answer #B: How many unique PvP types of game world servers are there currently in Tibia?

Solution code below:

SELECT COUNT(DISTINCT pvpType) AS PvP_Types_Number
FROM worlds

Answer #C: How many current worlds are there per PvP Type?

Solution code below: (in white letters)
SELECT pvpType
,COUNT(worldname) AS Number_of_online_worlds
FROM worlds
GROUP BY pvpType

Answer #D: How many worlds are there currently in Tibia per server location?

Solution code below:

SELECT servLocation
,COUNT(worldname) AS Number_of_online_worlds
FROM worlds
GROUP BY servLocation

Answer #E: How many current online worlds are there per server location and PvP type?

Solution code below:

SELECT servLocation
, pvpType
,COUNT(worldname) AS Number_of_online_worlds
FROM worlds
GROUP BY servLocation, pvpType
ORDER BY 3 DESC

Share

Share novsto

Subscribe now

The data analysis was done, mostly, using SQL, and a bit with Excel. The visualizations were done with Excel.

let's connect! ping me on LinkedIn

• Side note: Rumor has it that positions such as user acquisition manager, media buyer, and marketing analysts are relatively easy entry level positions for folks interested in being data analysts. I wasn’t one, but to know the basics of SEO, or Search Engine Optimization, is generally nice to know if you are active online.

Share

I believe it was around early April 2023 when Google emailed me a summary of my website's, s-rabi.com,2 Search Performance.3 This seemed like a nice opportunity to analyze some interesting data and take action accordingly to improve my website / website ranking on google. So I did it :)

After analyzing and discovering some insights, I took action and updated my website in order to rank better on Google and gain more traffic.

It’s not that my personal website has suddenly become an internet sensation, but there was certainly an improvement in terms of number of impressions, total clicks, average click-through-rate, and even average page-ranking position.

Subscribe now

Terminology

• Clicks is how many times a user clicked through into the website.

• Impressions is how many times a user saw a link to the website in the search results.

• CTR (Click-Through Rate) is the percentage of impressions that resulted in a click ([clicks/impressions]*100).

• Position is the average position of the website in the search results, based on its highest position whenever it appeared in a search.

Getting To Know The Data

I downloaded the web report for the maximum time: 24 Dec. 2021-24 Apr. 2023.

It was immediately clear that my first Medivia4 life-thread was the primary source of traffic.

| Clicks | Top Pages

| 478 | medivia-journey

| 8 | home page

| 3 | search/label/gaming

| 1 | Tutorial-How-to-Host-Free-WordPress-Website-Google-Cloud-Platform

| 1 | medivia-journey-lera-levels-201-300

| 1 | New-Old-Libya-R9X-Strike-Case-Revealed-US-Ramps-Up-Ninja-Bomb-Targeted-Killings

| 1 | welcome (about me page)

People from 42 different countries visited my website. The majority of them were from Brazil and Poland.

I scoped in on the data for an overview of the performance trend. The best scenario would have been to see a trend in which the Positioning trend and the CTR move towards one another.

• Position improvement = decreasing numbers, website appearing higher in the search results; and 

• CTR improvement = increasing numbers, more clicks per impression). 

But this was not the case, as can be seen in the figure below:

However, the data overall was not too bad as the website generally got more and more clicks.

This made me think that I needed to improve the SEO of the website because people were interested in its content. Yet there was a mismatch between their search words and the keywords used in my posts.

Therefore, I proceeded  to analyzing the Google users search queries made by the people who visited my website.

Subscribe now

Preparing The Data For Analysis

I downloaded the data as a CSV. Available here for you too <click here>.

CSV Stage

I first used the =LEN(...) to identify the strings length per query. 

I then used the =MAX(...) to identify the maximum length. 

That was used to determine the length of the variable named TopQueries. 

In this case, VARCHAR 55 since the lengthiest was 54.

Afterwards, I eliminated the percentage format for the CTR variable (%) and changed it to General. I then defined accordingly the CTR variable as DECIMAL(5,4).

Microsoft SQL Server Management Studio Table Creation

IF OBJECT_ID('Queries', 'U') IS NOT NULL
DROP TABLE Queries
------
------
CREATE TABLE Queries (
  TopQueries VARCHAR(100) NOT NULL,
  Clicks INT,
  Impressions INT,
  CTR DECIMAL(5,4),
  Position DECIMAL(5,2)
);
-----
-----
BULK INSERT Queries
FROM 'C:\Users\userName\Desktop\Practice set\GoogleSearchReport\Queries.csv'
WITH (
FORMAT='CSV',
FIRSTROW=2
);
------

SQL Data Analysis: Top Search Queries

In short, the idea was to identify the top-performing keywords to do search engine optimization (updating posts titles, metadata, etc.) by finding single keywords which were top-performing and to then find duo/bigram5 keywords which were top-performing as well.

Single Keywords Analysis

SQL Code

--Temporary table creation: saving single keywords and their metrics.
CREATE TABLE #SingleKeywordAnalysis (
    Keyword NVARCHAR(55),
    Clicks INT,
    Impressions INT,
    CTR DECIMAL(18, 4),
    Position DECIMAL(18, 4)
);

--Insert single keywords and their metrics into the temporary table.
WITH QueryKeywords AS (
    SELECT
        LTRIM(RTRIM(m.value)) AS Keyword
        , Clicks
        , Impressions
        , CTR
        , Position
    FROM Queries
    CROSS APPLY STRING_SPLIT(TopQueries, ' ') AS m
)

INSERT INTO #SingleKeywordAnalysis (Keyword, Clicks, Impressions, CTR, Position)
SELECT
    Keyword,
    SUM(Clicks) AS Clicks,
    SUM(Impressions) AS Impressions,
    SUM(Clicks) / NULLIF(SUM(Impressions), 0) AS CTR,
    AVG(Position) AS Position
FROM QueryKeywords
GROUP BY Keyword
;

--Select single keywords and the metrics, sorted by descending desired metric
SELECT
    Keyword
    , Clicks
    , Impressions
    , CTR
    , Position
FROM #SingleKeywordAnalysis
ORDER BY Clicks DESC, Impressions DESC, CTR DESC
;

--Drop the temporary #SingleKeywordAnalysis table
DROP TABLE #SingleKeywordAnalysis
;

Output

There were 326 different keywords. The below image contains selected part.

I have quickly went over the keywords in order to identify if there were any weird ones which I should avoid in advance in including them in the next part of my analysis. I found a number of them and made sure to exclude them for the next part.

Bigram Keywords Analysis

SQL Code

--Temporary table creation: saving bigrams and their metrics.
CREATE TABLE #BigramAnalysis (
    Bigram NVARCHAR(55),
    Clicks INT,
    Impressions INT,
    CTR DECIMAL(18, 4),
    Position DECIMAL(18, 4)
);

--Define ignore list (table with words to ignore).
DECLARE @IgnoredWords TABLE (Word NVARCHAR(55))
;

INSERT INTO @IgnoredWords (Word)
VALUES ('of'), ('on'), ('for'), ('at'), ('the'), 
('to'), ('by'), ('+ñ+É+Æ+£'), ('a'), ('+£+¬+¬'), 
('+¿+¢+û'), ('in'), ('+º+ñ+í+ò+£+ò+¬'), ('+á+ò+û+£+Ö'), ('+á+ù+£+ö')
;

--Generate bigrams and their metrics.
WITH QueryKeywords AS (
    SELECT
        LTRIM(RTRIM(m.value)) AS Keyword
        , Clicks
        , Impressions
        , CTR
        , Position
    FROM Queries
    CROSS APPLY STRING_SPLIT(TopQueries, ' ') AS m
    WHERE NOT EXISTS (SELECT 1 
                      FROM @IgnoredWords 
                      WHERE Word = LTRIM(RTRIM(m.value))
                      )
),
Bigrams AS (
  SELECT
  LAG(Keyword, 1) OVER (PARTITION BY Clicks, Impressions, CTR, Position       ORDER BY Id) AS FirstWord
 , Keyword AS SecondWord
 , Clicks
 , Impressions
 , CTR
 , Position
 , ROW_NUMBER() OVER (PARTITION BY Clicks, Impressions, CTR, Position ORDER BY Id) AS Id
 FROM (SELECT Keyword
       , Clicks
       , Impressions
       , CTR
       , Position
       , ROW_NUMBER() OVER (ORDER BY (SELECT NULL)) AS Id 
       FROM QueryKeywords) AS t
)

INSERT INTO #BigramAnalysis (Bigram, Clicks, Impressions, CTR, Position)
SELECT
    FirstWord + ' ' + SecondWord AS Bigram,
    SUM(Clicks) AS Clicks,
    SUM(Impressions) AS Impressions,
    SUM(Clicks) / NULLIF(SUM(Impressions), 0) AS CTR,
    AVG(Position) AS Position
FROM Bigrams
WHERE FirstWord IS NOT NULL
GROUP BY FirstWord, SecondWord
;

--Select bigrams and their metrics, sorted by descending desired metric.
SELECT
    Bigram
    , Clicks
    , Impressions
    , CTR
    , Position
FROM #BigramAnalysis
ORDER BY Clicks DESC 
     , Impressions DESC
     , CTR DESC
;

--Drop #BigramAnalysis temporary table.
DROP TABLE #BigramAnalysis;

Output

Combined Analysis: Last Analysis Step

I cut down the scope of the next part of the examination to single and duo- keywords with at least 250 Impressions.

I reviewed the results in Excel, manually, in order to gain a deeper understanding compared to the depth of understanding I anticipated I'd obtain by further SQL query analysis, such as union all for the two truncated tables and counting the frequency that the single keywords appeared within the bigrams. Et cetera.

Additionally, I removed the non-Medivia related keywords because Medivia proved to be where effort needed to be invested for the most impact on traffic.

Findings:

1. I was able to improve the SEO of my already existing post by adjusting the title, metadata and initial paragraphs of the post to include words such as Medivia Archer Guide, leveling guide, hunting places, faction progress, mining guide, et cetera.

2. It was clear I could use screenshots I saved from the progress of two other sub characters I had on Lera's account, a Mage6 and a Druid, to attract additional traffic given the top performing keywords.

3. I would be able to leverage the gained knowledge here in advance as I'd create my Knight Iron Man project life thread.7 This was a request I received two weeks ago anyway to create such a life thread.

Google email me once a while some funny achievements, statistics and other notes. The updates slowly but surely resulted with improvements to gaining organic traffic. About a month or so after taking actions, I got this, for example:

Ping me on LinkedIn

Share

Subscribe now

I then saw the dataset about Unicorns - StartUps with evaluation of USD 1+ Billion.3

I was curious if there was any relationship between Happiness and Unicorns.

Although I was quick to find out that there wasn't, I decided to follow up on some examinations given the even more interesting questions asked by Gallup4 since they survey folks around the world for the report, which I thought were very relevant for an environment resulting with successful startups.

But TL;DR— I found no evidence to suggest that. 

I did decide, however, to visualize and post it here because it was quite a challenge to decide how I should communicate these findings of "lack of correlation."

Insight from this practice:

In analytics, we often face data without clear correlations / significant correlation. In such instances, I advocate for minimalist, color-sparse visuals. Color should highlight key points, and here, the key point is the lack of correlation. Its absence would enable any presenter to convey the point forcefully.

I present the findings first & then note the SQL written to load and analyze this data (visualizations made with Excel).

Subscribe now

Visualizations

People were asked "are you dissatisfied with your freedom to choose what to do with your life?" with a binary response option of No (0) and Yes (1).

I expected to see people were more satisfied where more unicorns existed. But as shown below, that's not what the data suggests.

Similarly, I expected perceptions of corruption and confidence in the government to be favorable to unicorns. But such evidence was not found.

People were asked “Is corruption widespread throughout the government in this country or not?” and “Is corruption widespread within businesses in this country or not?” (Where data for government corruption are missing, the perception of business corruption is used as  the overall corruption-perception measure). 

Similar question was asked for confidence in the government.

Bottom line:

In analytics, we often face data without clear correlations / significant correlation. In such instances, I advocate for minimalist, color-sparse visuals. Color should highlight key points, and here, the key point is the lack of correlation. Its absence would enable any presenter to convey the point forcefully.

Subscribe now

Technical notes: The SQL written to load and analyze this data.

• Microsoft's SQL Server Management Studio.

----DROP TABLE: UNICORNS
IF OBJECT_ID('unicorns', 'U') IS NOT NULL
  DROP TABLE unicorns;

---CREATE unicorns
CREATE TABLE unicorns (
    Company varchar(50) NOT NULL,
	Valuation FLOAT NOT NULL,
	Industry varchar(100) NULL,
	Country VARCHAR(50) NOT NULL,
	ValuationDate DATE NOT NULL
);

---LOAD CSV unicornslist to load unicorns
BULK INSERT unicorns
FROM 'C:\Users\-----------\Desktop\Happiness and Unicorns\unicornslist.csv'
WITH (
FORMAT='CSV',
FIRSTROW=2
);

----
----DROP TABLE happiness
IF OBJECT_ID('happiness', 'U') IS NOT NULL
  DROP TABLE happiness;

---CREATE happiness table
CREATE TABLE happiness (
 CountryName VARCHAR(50) NOT NULL,
 RegionalIndicator VARCHAR(50) NULL,
 Year DATE NULL,
 LifeLadder FLOAT NULL,
 LogGDPPerCapita FLOAT NULL,
 SocialSupport FLOAT NULL,
 HealthyLifeExpectancyAtBirth decimal(5,3) NULL,
 FreedomToMakeLifeChoices FLOAT NULL,
 Generosity FLOAT NULL,
 PerceptionsOfCorruption FLOAT NULL,
 PositiveAffect FLOAT NULL,
 NegativeAffect FLOAT NULL,
 ConfidenceInNationalGovernment FLOAT NULL
);

---LOAD CSV happiness world report to load happiness
BULK INSERT happiness
FROM 'C:\Users\-----------\Desktop\Happiness and Unicorns\worldhappinessreport.csv'
WITH (
FORMAT='CSV',
FIRSTROW=2
);

------ Analysis Part
-- CTE Part
WITH numUniCo AS (SELECT Country
      , count(Company) AS unicornsNum
	  FROM unicorns
	  GROUP BY Country
),

hapi AS (SELECT CountryName
	  , AVG(LifeLadder) AS AVG_LifeLadder
	  , AVG(FreedomToMakeLifeChoices) AS AVG_FreedomToMakeLifeChoices
	  , AVG(PerceptionsOfCorruption) AS AVG_CorruptionPerception
	  , AVG(ConfidenceInNationalGovernment) AS AVG_GovtConfidence
	  , AVG(SocialSupport) AS AVG_SocialSupport
	  , AVG(Generosity) AS AVG_Generosity
	  , AVG(PositiveAffect) AS AVG_PositiveAffect
	  , AVG(NegativeAffect) AS AVG_NegativeAffect
FROM happiness
WHERE CountryName IN (SELECT Country
                      FROM numUniCo)
GROUP BY CountryName
)

---- Final Query
SELECT CountryName
	  , numUniCo.unicornsNum AS TotalUnicorns
	  , AVG_LifeLadder
	  , AVG_FreedomToMakeLifeChoices
	  , AVG_CorruptionPerception
	  , AVG_GovtConfidence
	  , AVG_SocialSupport
	  , AVG_Generosity
	  , AVG_PositiveAffect
	  , AVG_NegativeAffect
FROM hapi
JOIN numUniCo
ON hapi.CountryName = numUniCo.Country
ORDER BY 2 DESC

Decision makers repeatedly struggle with the dilemmas and perplexities of the principle of proportionality. It is an burdensome task to determine whether or not the attack’s anticipated military advantage is proportional or disproportional to the expected civilian harm (a.k.a. collateral damage), particularly during the often-inimical circumstances under which these decisions must be made. That is also the case in targeted killing operations, which afford relatively comfortable conditions. I have decided to confront this challenging problem because although it is old and known, it was without adequate remedy. I offer a data-based tool for decision makers to evaluate proportionality with. This tool resolves the core complexities in proportionality evaluation for Israeli targeted killings. In order to do that I have created a database with four attack types covering five years. The database maintains each attack in disaggregated form, and each such record contains several variables. The tool retrieves the most relevant direct military advantage that decision makers can expect to achieve by performing the operation. Minor instructions transform the regular assessment of expected collateral damage to the desired form. This platform sets the military advantage and the collateral damage in equivalent terms and in numeric form, and it thus frames an effective comparison. The tool’s flexibility increases functionality and maintains maneuverability for decision makers. It is intended to provide decision makers with a great point of departure to evaluate proportionality based on accumulated data and knowledge (on top of timely intelligence). Modestly put, it solves the majority of their dilemma.

Share

*Originally published: the Security Studies Review, by Shahaf Rabi, 07/06/2020.

Background

What’s Proportionality?

The principle of proportionality is one of the four core principles of the Laws of Armed Conflict, and it encapsulates the other three core principles in it. The principle, codified in the Geneva Conventions, prohibits to perform “an attack which may be expected to cause [civilian harm] which would be excessive in relation to the concrete and direct military advantage anticipated.” Therefore, decision makers need to assess what is the expected civilian harm (also known as collateral damage) and what is the anticipated direct military advantage (interchangeably called military necessity). The comparison of the two and the decision to follow is the proportionality evaluation.

Why This Problem?

The evaluation of proportionality is an onerous task, especially under operational constraints. There is no better person to testify about this hardship than Yuval Diskin, the Israeli Security Agency’s Deputy Director in 2000-2003 and Director in 2005-2011. He is probably the person who was involved in these precise decisions for targeted killing operations more than anyone else in Israel.

(Video below, The GateKeepers, in Hebrew, watch 1:25-3:20)

It is undoubtedly a problem when someone as highly experienced as Diskin battles with proportionality evaluation because if that is the case for him then what is of less experienced decision makers? Also, it is not as if bright minds did not attempt to solve this issue or create tools for decision makers. See for example: Guiora, Byman.

So What’s the Crux of the Problem?

In short, there are two main issues which complicate the evaluation of proportionality, especially for people who care about civilian lives:

(1) It is unclear how broad or narrow should the calculus of military advantage and collateral damage be. As such, the considerations are fluid and holistic since it is ambiguous what to include or exclude in each of them. See for example: Wittes, Blank, Schwartz, Cohen & Shany.

(2) The assessments are subjective – influenced by biases, heuristics, individual’s personality, culture, norms and values. Distinct people assign different weigh to the various elements which compose each of the competing interests (mil. adv. and coll. damage). For example, see: Whittemore, Cohen & Shany.

How I Resolved the Crux of the Problem

This issue bugged me long after I’ve been exposed to Diskin’s testimony. My light bulb moment came after I’ve read Ganor’s book chapter about the proportionality dilemma. He proposed a novel concept: quantified evaluation based on an objective calculation. However, I differed with most of the propositions, even for the more calibrated targeted killings one. It was simply not objective due to the assigned values into variables; the variables’ mathematical operations (e.g. X^2); and the overall calculations. Still, I realized the potential of numerical use in order to overcome the issue of subjectivity. I therefore returned to peruse the law with a numeric perspective in mind.

I sought out the agreed upon fundamentals regarding the above-mentioned first issue of holistic calculations of military advantage and civilians harm. The solid common ground concerns the basics of how states codified the treaty-law and the conservative interpretations of the text. The principle explicitly notes “concrete and direct military advantage anticipated”. That is, the reasonably foreseeable direct result of the attack. It is similarly agreed with regards to collateral damage that the top priority – which is also most tangible – is the physical harm to civilians (fatalities and physical wounds).

Hence, consider the simplified proportionality evaluation below in preparation for numerical evaluation:

• Proportional (attack may proceed):
  direct military necessity anticipated >= expected civilian harm.
  * though note the expected civilian harm may be greater than the anticipated direct military advantage for as long as it is not “excessive”. Then again, this would have us circle back to issues of subjectivity.

Importantly, the operations conducted as a part of Israel’s targeted killing policy deal with the ones which serve the purpose of thwarting specific Palestinian attacks that the security services obtained information on before the attacks materialized.

I figured that as the purpose is to prevent attacks, then this is the perfect fit for direct military necessity anticipated. There is a rich historical record of Palestinian attacks and their respective results in terms of fatalities and physical injuries. So I created a dataset with the details of individual attacks (especially the attack types and casualties). Therefore, I solved the first issue of holistic calculations since both, the military necessity and civilian harm, are expressed in terms of fatalities and physically wounded people.

Moreover, this actually also solves the subjectivity issue. The reason for this is that the database records physical casualties for the military necessity. These are facts, objective numbers. Similarly, the expected civilian harm is also based on assessments of the number of fatalities and injuries. Consequently, the anticipated direct military necessity in terms of foiling an attack by performing a targeted killing is in the form of objective numbers, and so is the expected collateral damage.

The platform therefore frames an effective comparison as it sets the military advantage and the civilian harm in equivalent terms and in numeric form. The most basic of intelligence information about the impending attack, the modus operandi, already enables this data-based tool to retrieve the most relevant* direct military necessity that decision makers can expect to obtain if the targeted killing is carried out in order to thwart the threat. With this as anchorage point, the only possible adjustment needed to compare the military necessity with the civilian harm is to ensure the regular assessment of expected collateral damage relates to physical casualties. At minimum, this quick and simple solution solves decision makers’ most difficult part in proportionality appraisal. However, it neither replaces their discretion nor determine for them if the proposed attack is proportional or not.

*Relevant as in the tool retrieves values based on their percentile location. It avoids the use of the average which is susceptible to extreme cases. Further adjustments depend on available intelligence information (timely and past research – as exemplified later on).

INSIGHTS (sneak peek)

Important acronyms:

1. for Modus Operandi variable:
   1.1. SMS (Suicidal Mass-Shooter)
   1.2. SB (Suicide Bomber)
   1.3. VBED (Vehicle-Borne Explosive Device)
   1.4. SB VBED (Suicide Bomber via Vehicle-Borne Explosive Device)

2. for De-Facto Target variable:
   2.1. PT (Public Transportation)
   2.2. SF (Security Forces)
   2.3. O (Other)

Overview of the Data

DATA ANALYSIS INSIGHTS – For Smarter Usage

The data analysis provided opportunities to identify KPIs (key performance indicators) since the established database contains a number of cataloged variables, and some of them proved relatively insignificant. This sneak peek focuses on the two most important indicators: the modus operandi; and the de-facto target.

How many times each modus operandi targeted Other, Public Transportation & Security Forces

How many casualties (fatalities/injuries) for each modus operandi, sorted by target (Other, Public Transportation, Security Forces)

It is useful to gain insights from data analysis for smarter military necessity assessment. For instance, there are attack types such as the VBED with 46 incidents in total, a fairly small number. This might become a problem if intelligence points to an organization like Hamas that is about to stage a VBED attack. If filtered by Hamas, 3 incidents are left in total. Yet after in-depth analysis, there is no evident difference between attacks’ lethality when comparing organizations. It means the assessment should not refine the military necessity anticipated by organization in this case. However, the fact Hamas facilitates the attack is important because Hamas overall inclines to target public transportation and other – a KPI.

DATA ANALYTICS INSIGHTS – Knowledge/Data-based Direct Military Necessity Anticipation

The decision makers need to have at their disposal a platform that is informative and easy to comprehend. The intelligence services need to make available the concrete intelligence information for the impending attack as well as supplementary intelligence analysis reports in order to determine the layers upon which the assessment of the anticipated direct military necessity is to be based on. The KPIs constitute the backbone of the assessment. Further intelligence analysis can help indicate other patterns, preferences, et cetera which may be relevant to the assessment (e.g. previous example of Hamas’ target choice tendency). On top of this, the concrete intelligence information need be put, including possibly, recommendations in advance to opt for a certain percentile range. For instance, to opt for 50% percentile range and strongly recommend refraining from increases to a certain degree in cases of modus operandi with thousands or more of cases recorded in the database, as opposed to modus operandi of much smaller recorded cases.

An approachable user-interface greatly matters. The platform itself clearly visualizes to the decision makers the percentile range as well as indicate the spread of the values since they can see the median and the values at each edge of the percentile range. These help widen their perspective, as exemplified below (reflects the overall direct military necessity anticipated in case of suicide bomber attacks).

Direct military necessity anticipated for suicide bombers

Finally, the decision maker(s) need to know the assessment with regards to the expected civilian harm, which might also be given as a range depending on the circumstances at hand.

Proportionality Evaluation – Uncomfortable & Unintuitive Cases in Point:

Let us take some inspiration from Yuval Diskin’s testimony about proportionality evaluation. You are the commander in charge in the three given scenarios. Your final decision, should you make one, matters less than your ability to make rapid, informed, and sagacious proportionality evaluations.

(1) You finally know the cell phone number that the PIJ’s suicide bomber carries with him. This enables you to pinpoint his location, and you find out the bomber is traveling in a taxi. You have no clue how many people are in it, although you assess it is a civilian taxi driver at minimum and up to additional 3 other civilians. You know the bomber was told to explode at one of the city’s most popular streets with many bars and restaurants, and also, not to waste time. The taxi travels through a quiet street, seems without people. There is no time to pass on the information to security personnel on the ground before the bomber reaches his destination. The line is open with the IAF’s command and control operations room. They see what you see. The drone is ready to fire a missile. Whoever is in the taxi is expected to be killed. Is it proportional to strike? (which does not necessarily mean you’d order the airstrike. You may choose to let the bomber go ahead, and probably explode while you cross your fingers for a malfunction of any sort of the explosive belt).

Direct military necessity anticipated for a suicide bomber targeting “Other”

(2) IAF drone tracks a suicide bomber in the West Bank. He travels in a Palestinian taxi with the driver and another unknown Palestinian. As the commander, you hope to wait for a moment of opportunity to strike without any uninvolved people around. However, you see the taxi is about to arrive to a flying-checkpoint manned by five IDF soldiers. The soldiers are unaware of the danger and cannot be warned in time. Do you conduct an airstrike at the Palestinian taxi or see how things play out at the checkpoint?

Direct military necessity anticipated for a suicide bomber targeting “Security Forces”

(3) You located the suicide bomber at a junction with two unknown people, most likely Israeli civilians. They ordered one of Dan’s Bubble minibuses (i.e. 10 seats max.) and it is about to arrive to them in no more than two-three minutes. This is a Hamas suicide bomber, and thus you assess she is likely to explode in it. While the other personnel under your command do their best to find out with Dan how many passengers are currently in the minibus, and to contact the driver, you need to consider whether you should act in case the efforts won’t bear fruits in time.

Direct military necessity anticipated for a suicide bomber targeting “Public Transportation”

SOME TECHNICAL NOTES

(1) I knew from previous researches that there was no adequately comprehensive, reliable, and publicly available dataset of Palestinian attacks. It had to be created. I thus designed and used the methodology and coding rules document for the Palestinian & Co. Political Violence database (hereinafter, database).

(2) The reports of the ISA (aggregated summaries) proved useful for the purposes of choosing which kind of attack types to prioritize and to collect due to time and other constraints.

(3) I created the database. Subsequently, I imported the 357 relevant attack cases to a new Excel file, and in Excel’s Manage Data Model/Power Pivot, I further prepared the data for analysis. For instance, I removed redundant cataloged variable-columns for the purposes of this analysis.

(4) I conducted analysis in order to identify KPIs and gain additional insights. This required to add some DAX at the calculations area, such as the retrieving the fatalities and injuries values in the right percentiles. Moreover, I used the Power Pivot functions to re-arrange the data and to create new tables for some additional customized investigation of the data, to visualize the data, and so on, in order to gain further insights.

Subscribe now

Brief Backstory

It was early June 2019 when I finally read the WSJ’s article. It was first to confirm the existence and use of the R9X missile. The WSJ3 stated the R9X was employed “about half a dozen times” in Iraq, Libya, Somalia, Syria and Yemen, before 9 May 2019. However, out of the about six times in which the missile was used, the article only identified two specific cases: the 26 February 2017 targeted killing of al-Masri in Syria; and the 1 January 2019 targeted killing of al-Badawi in Yemen.

My focus and expertise were, and still are, on Israeli targeted killings, not U.S. ones. Still, I entertained myself with the idea of finding the unidentified strikes. My colleagues, Professor Avery Plaw (zl) and Carlos Colon,4 were tracking U.S. targeted killings, but they told me they weren’t going to search for it because their hands were full. I thought I could invest several hours searching for the cases during the garbage time (very late in the day, when I was less productive – especially in considering these were days I’d wake up around 5:30 and work until 20-21:00 on various projects and tasks, among them, attempting to secure a spot in a Political Science PhD program in one of the American Universities I’ve set my eyes on). Therefore, I decided to dig into their U.S. targeted killings database.

Reconstructing in Hindsight the Data Analysis & Research Process

Clearly, the WSJ’s article noted about six strikes in five countries which occurred before May 2019. It also identified two cases, one in Syria and another in Yemen.

Filter data by Country: I worked under the assumption that about six strikes meant 5-7 cases as well as that no country was named without at least a single R9X strike. I was not able to tell if, and if so which, country had more than one strike. It was thus best to focus on the remaining countries, namely: Iraq, Libya and Somalia. This was important because my colleagues did track US targeted killings in, for example, Yemen. I opted out of going through the many Yemeni records. However, my colleagues did not track US targeted killings in all three remaining countries, so I only had data for Somalia and Libya.

Filter data by Date: As simple as it sounds, I filtered out data after 9 May 2019 (WSJ article publication). As for how far back to go, I perused the WSJ article for hints. Note, for example, “The weapon was under development as early as 2011” (my emphasis) and also “Development and refinement of the weapon drew impetus from Mr. Obama’s policy announcement in May 2013 that set new rules for using lethal force […]” (my emphasis). This suggested there was no reason to go back in time more than early 2011, although likely that May 2013 would have been enough.

The two above steps weren’t sufficient because there were still too many cases to go through for a low priority project. Yet the sources of the WSJ journalists basically hinted that the use of the R9X was especially designed for targeting and hitting vehicles in order to minimize civilian fatalities. Consequently:

Filter data by “Object_Type” variable: I decided to analyze the dataset by the Object_Type variable. I kept the records if the Object_Type variable was equal to “Vehicle”.

Filter data by “Fatalities” variable: I next believed it would be efficient to review the data by paying attention to the number of fatalities. Perhaps it was optimism, or just a late in the day misjudgment when my critical thinking was on low alert, but I chose to filter the data again, keeping only cases of 0-2 fatalities (hoping the U.S. was successful). In hindsight, this might’ve well been a mistake and I should’ve kept 3-5 fatalities (bearing in mind number of people in most vehicles and option of harm to nearby people). I then re-arranged the data by fatalities in ascending order.

* * *

Finally, I began to go through the records case by case. I read the text within the “Incident_Summary” variable, looking for clues and to get my head around each incident. This helped discard and move on from some cases.

However, I mostly had to look into each of the individual sources with which my colleagues processed before synthesizing the main/final record. In other words, a number of sources were processed for each data-record. In there, I quickly inspected archived links which often contained photographs from the scenes of the strikes (if there weren’t any then I’d spend a few minutes searching externally).

Importantly, in the imagery from the strike’s scenes, I looked for telling indicators regarding the type of munition which was used in the targeted killing strike. As the R9X missiles do not have an explosive warhead, but blades, the impacts cause distinct signs. For instance, lack of any burnt parts of vehicles in the absence of explosion, the caved-in top of vehicles (often with unique shape due to the missile’s blades), and so on.

* * *

I began going through the records only around July 2019. I believe it was on the second or third day when I found the Libya strike (spending no more than 2-4 hours). After a few more days, I told my colleagues that I found the Libyan case but not the Somalian one.

Although it was fun and nice-to-have, as academically-minded people the scoop itself was not really valuable for us, particularly as there were not enough cases to compare aspects of operations, such as collateral damage between this new munition and others munitions or methods. So I just stored this discovery and forgot about it.

I saw the rise of R9X missiles usage in June 2020. It reminded me of this scoop/discovery. I was surprised nobody else identified the Libya case (nor the Somalian/Iraqi for that matter) since an entire year has passed. I hence decided to share my finding. That noted, I nonetheless argue the R9X is no silver bullet5 when it comes to minimizing or avoiding altogether civilian casualties. It has yet to prove itself.

6

Subscribe now

Floyd was killed in Minneapolis, a city with an official and horrible record4 of police violence against Black people. In the past five years, about 57% of all police violence was against them although African-Americans composed only around 19% of the city’s entire population. For comparison, consider the fact White people composed 60% of Minneapolis’s population and their share of police violence was only 24%.

However, the riots are nationwide, so I explored the broader story of the fatalities by U.S. law enforcement. I retrieved data from the Centers for Disease Control and Prevention,5 U.S. Census Bureau6 and Fatal Encounters.7

First step to gain perspective: Focusing on the three largest race/ethnicity groups based on the 2010 U.S. census (which compose 92% of the U.S. population together) and their respective amounts resulting in deaths due to encounters with U.S. law enforcement personnel.

First step to gain perspective: Focusing on the three largest race/ethnicity groups based on the 2010 U.S. census (which compose 92% of the U.S. population together) and their respective amounts resulting in deaths due to encounters with U.S. law enforcement personnel.

Figure 1:

Moreover, age is a key factor which reflects the odds of being killed by the police.

Paying attention to the differences between the fatalities trend of African Americans and the European Americans given the overall image/understanding we have. Notice the not-so-subtle differences.

African Americans have it most dangerous in their 10s and 20s. The trend rapidly declines afterwards. However, for European-American, the peak arrives during their 30s and declines much more gradually in comparison to the two other groups.

I suppose this might well be the result of socio-economic gaps between the groups.

Figure 2:

Furthermore, as shown in Table 3, gender is another key indicator of the likelihood that a person be killed. This is regardless of the age group or race/ethnicity. Males are far more likely to be killed in comparison to females.

Figure 3:

The only good news seems to be that African American females do not seem to be killed more than European American females (though bearing in mind it is still disproportionate to the general population).

Figure 4:

Examination of the causes of death for males between the ages of 10-49 exposes no clear difference based on race/ethnicity, although it raises the suspicion that U.S. police uses disproportional force against African-Americans compared to European-Americans. It results with more fatalities by tasering, asphyxiation, beatings, and pepper spraying. Table 5 displays this data (containing 6 out of 17 most ‘interesting’, cataloged causes of death).

Figure 5:

Sadly, this is just the tip of the iceberg with regards to how police violence hurts African-Americans. For instance, the primary finding of a 2018 research8 about the “spill over effects [of police killing] on the mental health of black Americans” shows it harms their mental health. This, in turn, effects suicide rates, and so on.

NOTE: 24 AUGUST 2023:

I’ve chosen to re-post this brief old data analysis project I’ve done when I polished my Excel skills back in August of 2020 (power pivot, using the data model, visualizations with Excel, etc.) because it nicely reflects the improvement that can be achieved later on, particularly with regards to visualizations and conveying a strong message alongside them. Anyone with a sharp eye can spot the problematic titles, missing legends, etc.

A wonderful resource for improving your visualization skills, particularly with Excel, is this website: https://www.storytellingwithdata.com/

Subscribe now

Dive into a digital realm where research and data analysis dance together with decision making, where data shifts from tables into forceful visualizations, and where a budding research analyst evolves into a seasoned data analyst.

This isn't just my portfolio, but a chronicle of my exploration into the expansive world of data analytics & decision making. Whether you're an aspiring analyst, a seasoned pro, or simply someone with a penchant for learning, this is your hub!

Here, I share the nuggets of knowledge I uncover, tips to enhance your analytics journey, and best practices that refine our collective expertise. I invite you to share your insights, ask questions, and spark enlightening discussions. We’ll grow as a community of practice. I envision the insights of experienced analysts enriching our collective knowledge and become better professionals. So join in! It’ll be fun :)

Subscribe now

Please make sure you read the About us page.

In essence, the vision is for Novsto to become a publicly available hub for reliable and high-quality research and analysis case studies and questions.

Share

• The materials are freely available, regardless if one is subscribed or not.

• Free subscribers have access to nearly all of the resources. Occasionally, members (paid subscribers) receive exclusive access to unique materials.

• Membership/paid subscriptions and/or gift donations are first and foremost forms of expressing support which help cover expenses.

Subscribe now

Give a gift subscription

This initiative was created by, and is managed by, Shahaf Rabi.

Google Scholar

SSRN