
[iframe {title: "MSDS - Capstone Project"}](https://joeyloganpython.github.io/Capstone/)


Capstone



# Fake News Detection in PySpark

Fake news is an article contains inaccurate and misleading information aiming to change one's opinion,
and in doing so gain power. In this work, I propose ML model to classify news into fake and reliable using PySpark

***

## Dataset {#dataset}

The news dataset was found at [Kaggle](https://www.kaggle.com/c/fake-news/data) contains three attributes (author, title and text) and the target column (0 - reliable, 1 - fake). There are 20,800 rows, each of which describes one article.
Also, there are 4,194 unique authors in this dataset.

---

## Exploratory Data Analysis (EDA){#eda}

#### Class Balance {#calss-balance}

A data set that contains significantly more instances of one class than other classes is considered to be im- balanced.
In such case, a machine learning algorithm might tend toward the major class.
Therefore, our first step is examining class balance as shown below:

![Class Balance {859×335} {caption: test}](fake-news-detection/dist.png)


The figure above shows the distribution of the target column.
There are 10,387 (49.93%) reliable articles and 10,413 (50.07%) unreliable articles.
Therefore, our data set is balanced.


#### Missing Values {#missing}

Missing values could lead to several errors that might lead to either termination of the program, or unreliable results.
Therefore, it is required to identify rows that are missing attributes as well as determine a proper way to deal with them.

There are 1957 rows that are missing the author attribute, 1931 are also labeled as fake.
Title is missing in 558 rows, all of them have both author and text and are labeled as fake.
Finally, text is missing in 39 instances, each of which is also missing the author at- tribute, and is labeled as fake.

To sum up, more than 98% of the rows that are missing one or more attribute are labeled as fake.
This will be helpful in creating a good baseline for the machine learning model.
Additionally, all 39 rows that are missing both the text and the author will be deleted because there is no content to process in those instances.


---

## Methodology {#methods}

#### Baseline {#basline}

A baseline is a guess that could be done easily by anyone,
and which the machine learning model is trying to improve.
Based on the EDA, one can label all rows that have at least one missing attribute as fake.
Otherwise, it is labeled as reliable.
In this case, precision is great since reliable was guessed for the majority of rows in our dataset.
However, the recall is low (0.56), so the accuracy score is 0.61 and F1 score is 0.77 as shown below:

```python
df_dummy = (spark_df
                .withColumn("prediction", 
                    when((col("author") == "NaN") | (col("title") == "NaN") , 1.0)
                    .otherwise(0.0))
                .withColumn("label", col("label").cast(FloatType()))
                .select(["label","prediction"])
            )
```

![Dummy Classifier  {250x200} {caption: Dummy Classifier}](fake-news-detection/dummy.png)

Our goal is to design a pipeline and utilize a machine learning algorithm
to classify news into either reliable or unreliable,
and achieve results that are better than the baseline.

#### Data Pre-Processing{#pre-processing}

Target class is balanced, but there are several rows that are missing at least one attribute.
Therefore, we must clean the data before developing pipeline.


![Cleaning Dataset {341x81} {caption: Cleaning Dataset}](fake-news-detection/pre-process.png)

First stage is deleting all rows in which the text is shorter than 60 characters.
The goal is to classify news articles, so instances with little to none content are not part of this study. Next stage deal with rows that do not have a title.
In this case, ”NaN” string is replaced with white-space.
As long as there is some content, it is valuable for the analysis.

Third, symbols (e.g. $,%,#) were removed from every title and text in each row.
Later on, text will be converted to numerical values, so it is an important step.
It prevents those characters from expanding the dimension of the variables, thus affecting machine learning model.

Finally, a new column is created, and is composed of both the title and the text.
There are some rows in which the text already includes the title.
In such cases, the text was taken.

#### Pipeline {#pipeline}

Document classification requires converting sentences to words, and then to numerical values.
The process contains 4 stages as described below. They are being executed one after the other
in order to represent each word as a number (frequency).

![Pipeline {341x81} {caption: Pipeline}](fake-news-detection/pipeline.png)

Tokenizer is the first stage in which the text is being split into a list of words.
It uses white-space as the splitter, and also convert upper case letters to lower case. Then, stop words were removed from each list of words.
Stop words are words that appear many times in a document, and have no significance for the analysis.
The list of stop words was obtained by using the [NLTK](https://www.nltk.org) package in Python.

The following stage is the Stemmer that converts every word to its stem.
This is a [custom transformer](https://csyhuang.github.io/2020/08/01/custom-transformer/) that utilizes PorterStemmer instance from NLTK package.
This step reduces the dimension of the features3 column.
For instance, the words ”Playing”, ”Plays”, ”Played”, and ”Play” are all converted to ”play”. Final stage is the Term Frequency (TF) in which the program count the frequency of every term in a document.
The result of this stage is our features column for the machine learning model.

```python
# Stage 1 - Tokenizing words
tokenizer = Tokenizer(inputCol="full_text", outputCol="full_text_words")

# Stage 2 - Removing stop words (using nltk stop words)
word_remover = StopWordsRemover(stopWords = stopwords_ls,
                                inputCol = "full_text_words",
                                outputCol = "full_text_words_clean")

# Stage 3 - Lemmatizing each word using custom lemmatizer class
stemmer = Stemmer(inputCol = "full_text_words_clean", outputCol = "stemmed")

# Stage 4 - Term Frequency of every word
tf = CountVectorizer(inputCol="stemmed", outputCol="features", vocabSize = 1e6)

pipeline = Pipeline(stages= [tokenizer, word_remover, stemmer, tf]).fit(train)
train_df = pipeline.transform(train).select(["full_text","features","label"])
test_df = pipeline.transform(test).select(["full_text","features","label"])
```


---

## ML Model {#ML}

The dataset was split 70% training and 30% testing.
After splitting the data, the designed pipeline was fitted on the train data, and transformed both training and test sets in order to compute the features column. In this study, Naive Bayes Classifier was utilized in order to classify news.
There are three types of Naive Bayes: Multinomial, Bernoulli, and Gaussian. The first one can handle finite discrete data,
the second can only handle binary (0,1) vectors,
and the last one can handle continuous data 4.
The features column contains numbers corresponding to the frequency of every term in the dictionary,
and therefore Multinomial Naive Bayes will be utilized.

---

## Results  {#results}

![results {250x200} {caption: Final Results}](fake-news-detection/results.png)

Precision was down by 0.2 compare to the baseline.
However, recall was significantly improved from 0.56 to 0.885, and therefore both accuracy and F1 score are much better than in the baseline.
To be more specific, Accuracy jumped from 0.618 to 0.921 and F1 score changed from 0.723 to 0.927.
F1 score can be viewed as the harmonic mean of precision and recall, thus proving this model’s reliability.

---

## Conclusions {#conclusions}

Recently, fake news has become ubiquitous in the media, leading to increasing levels of distrust amongst consumers.
To help mitigate the spread of fake news, this paper presents a pipeline to classify news articles.
This program was developed in PySpark, the optimal platform for processing big data.
Implementation of this program will greatly decrease the prevalence of fake news, thus improving news quality and trust amongst consumers.



Fake News Detection (PySpark)


[iframe {caption: "Floods visualization using Tableau Public"}](/floods/floods-visualization.pdf)

Floods Visualization


[iframe {title: "Insider Trading Alerts"}](https://avivfaraj.gitlab.io/insider-trading-alerts)


Insider Trading Alerts


# Mocking Techniques (Unit Test)

Unit test is a great tool which allows the programmer to quickly run multiple tests on the code and ensure that pushing new code doesn't break it. Writing unit test can be challenging in some programs such as database clients due to accessbility issues. In those cases, one can utilize the built-in `unittest.mock` library which allows replacing parts of the system with a mock object to make assertions and ensure the logic is correct.

***

## setup {#setup}

[SQLiteClient](/Projects/CS/context-manager) is tested using Pytest and the built-in `unittest.mock` library, so it is required to install `pytest`. Also, a coverage report can be generated by installing the `pytest-cov`, which is recommended, but is optional. Coverage report shows the percentage of the code that is being tested, and if some lines are missing, it prints them as well, so it can be useful to ensure full coverage.

---

## Pytest Fixtures {#fixtures}

Test fixtures can be accessed by test functions, and they provide consistent and reuseable baseline objects which can significantly reduce test code. They can also be utilized within methods of a test class. Both options are shown below.

{gap: 30px}
### Usage in Functions

Test functions can easily access Pytest Fixtures by simply adding them as parameters in the function signature as shown below:

```python
import pytest
from mocking.src.sql import SQLiteClient
from unittest.mock import patch


# Define a pytest fixture which can be used by test functions
@pytest.fixture
def db_client():

    # Patching the connection - can be ignored for now
    with patch("mocking.src.sql.sqlite3.connect") as mocked_connection:

        # Establish connection to database.
        # Since it was mocked, the connection will not be established,
        # but tests can still be executed to ensure correct logic.
        with SQLiteClient("test.db", timeout=20) as conn:

            # That's the value that will be passed to functions
            # that utilize `db_client` (listed as a parameter).
            # In this case, its a tuple!
            yield mocked_connection, conn


# Test Function - Note that the parameter db_client
#                 has the same name as the fixture above.
def test_connection(db_client):

    # Unpacking the tuple
    mocked_connection, conn = db_client

    # Running the test to ensure correct arguments while calling method.
    assert mocked_connection.called_once_with("test.db", timeout=20)
```

The example above uses `unittest.mock.patch` to mock the connection and prevent a db file form being generated, but it will be explained later on. Note that the name of the fixture, `db_client`, was passed as a parameter to the test function, `test_connection`. Pytest will automatically assign that value, and will execute the assertion to ensure that the function was called with the correct arguments.

{gap: 30px}
### Usage in a Test Class

There are two ways to utilize pytest fixture within a Test Class, both are explained below.
1. External Fixture - One must register the variables (using `request` as a parameter to the fixture) and use a special decorator `@pytest.mark.usefixtures` before test class definition in order to allow class methods to recognize and access the fixture. Consider the following example:

    ```python
    import pytest
    from mocking.src.sql import SQLiteClient
    from unittest import TestCase
    from unittest.mock import patch


    # Creating a SQLiteClient fixture to be used in the methods
    # of the SQLiteClientTestCase class.
    @pytest.fixture
    def mocked_connection(request):
        """
        Mocking connection to database. Note that since tests are
        done using a testing class, then `request` must be included as
        a parameter. The usage of request is necessary in order to
        define attributes that will be used inside the class.

        For example, `request.cls.mock_conn = ...` as appears below
        enables calling this parameter as an attribute:
        `self.mock_conn.assert_called_once()`

        Note: `yield` is used instead of return because
            return would have ended the `with` clause
            which would delete the mocking instance.
        """
        with patch("mocking.src.sql.sqlite3.connect") as mocked_connection:
            with SQLiteClient("test.db", timeout=20) as conn:

                # Connection mock instance.
                # Is used to test calls and arguments
                request.cls.mock_conn = mocked_connection

                # Connection instance.
                # Is used to execute SQLiteClient methods
                # so testing can be done before and after calling.
                request.cls.conn = conn
                yield


    @pytest.mark.usefixtures("mocked_connection")
    class SQLiteClientTestCase(TestCase):

        # Testing Attributes
        def test_attributes_1(self):
            """
            Testing pytest fixture attributes
            """
            assert self.conn.name == "test.db"
            assert self.conn.timeout == 20
    ```

    Note the usage of `request` within the fixture. It appears as a parameter to the fixture named `mocked_connection` and is being used to register class attributes by assigning a value in the following format `request.cls.ATTRTIBUTE_NAME =...`. In our case, there are two attributes, `mock_conn` (`request.cls.mock_conn =...`) and `conn` (`request.cls.conn =...`).

    Then, the test class `SQLiteClintTestCase` is connected to the fixture using the decorator `@pytest.mark.usefixtures(FIXTURE_NAME)` which allows us to use the registered attributes: `assert self.conn.name == "test.db"`.

2. Internal Fixture - In this case, the fixture is defined as a method inside the scope of the test class as shown below:

    ```python
    class SQLiteClientTestCase(TestCase):

        @pytest.fixture(autouse=True)
        def setup(self):
            with patch("mocking.src.sql.sqlite3.connect") as mocked_connection:
                with SQLiteClient("test.db", timeout=20) as conn:

                    # Connection mock instance.
                    # Is used to test calls and arguments
                    self.mock_conn = mocked_connection

                    # Connection instance.
                    # Is used to execute SQLiteClient methods
                    # so testing can be done before and after calling.
                    self.conn = conn

                    yield

        # Testing Attributes
        def test_attributes_1(self):
            """
            Testing pytest fixture attributes
            """
            assert self.conn.name == "test.db"
            assert self.conn.timeout == 20
    ```
    Note the use of `autouse=True` which means that the fixture will automatically be executed before each test method, so `mock_conn` and `conn` will be defined as class attributes, and can be used for assertions.

In both cases, note that the `yield` was utilized. The reason is that `return` statement would have closed the SQLiteClient session, so `conn` would be defined as `None`. `yield`, however, continues to the end of the test method.

{gap: 30px}
## Mocking {#mocking}

### Find the path

Looking at the examples in the previous section (`with patch("mocking.src.sql.sqlite3.connect") as mocked_connection`) one can find the usage of the following path: `mocking.src.sql.sqlite3.connect`, however, looking at the package structure, and file's content, there is no such path. In other words, folder structure is: `mocking.src` and the file name is `sql.py`, so that explain the beginning of the string: `mocking.src.sql`. In the import statements there is only one relevant statment: `import sqlite3`, which expands the string: `mocking.src.sql.sqlite`.

The last component (`connect`), appears inside a method that we would like to test, but we can't without mocking. To put it simply, the method `__enter__` tries to connect to a `sqlite3` database when it is invoked (using `with` clause). However, if the job is ran in a server with no database access, e.g. gitlab/github pipeline, then the job will fail. To avoid such scenario, connection mocking is required, and is applied to the following line:[`self.__conn = sqlite3.connect(self.name, timeout=self.timeout)`](https://gitlab.com/avivfaraj/pytest-mocking-examples/-/blob/main/mocking/src/sql.py?ref_type=heads#L67).

That's one way to find out the correct path to mock an object. Another way is utilizing the function `dir`. In the case discuss above, one could print the methods of `sqlite3` with the following lines:
```python
from mocking.src import sql

print(dir(sql.sqlite3))
```
One of the printed options is `connect`, and that's what we were looking for.

{gap: 30px}
### Patch

Patch is utilized to mock the object:

```python
from mocking.src.sql import SQLiteClient
from unittest.mock import patch

with patch("PATH") as mocked_object:

    # Execute code
    ...
```
`mocked_object` is used to run assertions on the object under test. There are several assertion methods available, among them are: `assert_called_once`, `assert_called_once_with`, and `assert_called_with`.

One can print available assertion methods by using the `dir` function: `print(dir(mocked_object))`. If used in pytest, then run pytest with `-s` option which tells pytest to show print statements.

Also, one can mock a function of a mocked object. Here is an example taken from SQLiteClientTestCase class:

```python
def test__exec_commit(self):
    """Testing CIUD"""
    self.mock_conn.assert_called_with("test.db", timeout=20)
    mock_cursor = self.mock_conn.return_value.cursor()

    self.conn._SQLiteClient__exec("TEST QUERY", parameters=(), rows=0)

    mock_cursor.execute.assert_called_once_with("TEST QUERY", ())
    self.mock_conn.return_value.commit.assert_called_once()
```

Above, the `mock_conn` was setup by the pytest fixture, and an `assert_called_with` was executed to ensure arguments passed to the `connect` call are correct.
Then, `mock_cursor` is mocking the `conn.cursor()` object, with the help of `return_value`. In order to test `mock_cursor`, a method is executed by using the connection object (`self.conn`). Note that `__exec` is a private method, so to access it one must add (`_SQLliteClient`). Finally, `mock_cursor.execute` is tested using `assert_called_once_with`. Additionally, `conn.commit()` is tested by asserting that it was called once using the `mock_conn` object. Syntax is a bit tricky, and I find `dir` function to be very helpful in understanding the scope of variables within a function.

{gap: 30px}
#### Patching with return value

Sometimes testing return statements is required to ensure logic is correct. For that, one can utilize the `return_value` argument in patch:

```python
@patch("mocking.src.sql.SQLiteClient._SQLiteClient__exec", return_value=[1, 2])
def test_read_execution_3(self, mock_exec):
    """
    Testing execution of `__exec` method
    when calling `read` method with return value set
    for the mocking in order to test the return statement
    """
    test = self.conn.read("SELECT * FROM TABLE WHERE name = ?", ("test",), 2)
    mock_exec.assert_called_once_with(
        "SELECT * FROM TABLE WHERE name = ?", ("test",), 2
    )
    assert test == [1, 2]
```

The example above is also taken from the SQLiteClientTestClass, but the fixture was not used. Instead, a new patch is used to test the returned value. Also note that the path to patch is different the fixture defined previously. The return value is set to `[1, 2]` which means that the `__exec` method will return that list, which should be returned from `read` as well, and that's what the assertion is testing. Once again, the `__exec` method was defined as private (using double underscore `__`), so in order to patch it, one must add underscore and the name of the class (`_SQLiteClient`) before the name of the method.

{gap: 30px}
#### Patching with side effect

`side_effect` argument can be used to raise errors, and test error logic. Simply setß `side_effect` to the desired error, and it will raise it, so the logic in the scope of the `except ...` clause can be tested:

```python
@patch("mocking.src.sql.SQLiteClient._SQLiteClient__exec",
       side_effect = ConnectionError("Testing logic when ConnectionError is raised!"))
def test_a():
    ...

```

## Example

A test class was developed for the SQLiteClient, and can be viewed in the [gitlab repository](https://gitlab.com/avivfaraj/pytest-mocking-examples/-/blob/main/mocking/test/test_sql.py?ref_type=heads). Coverage report was printed as part of the pipeline, and it shows 100% coverage:

![Coverage Report {450x139} {caption: Coverage Report}](mocking/coverage-report.png)

To create & print coverage report simply run the following commands:

```bash
python3 -m pytest --cov-report=html --cov=PATH_TO_SOURCE_FOLDER
python3 -m coverage report -m
```



Mocking (Unit Test)


[iframe {title: "Weather Alerts"}](https://avivfaraj.gitlab.io/weather-alerts/)


Weather Alerts


[iframe {caption: "Clustering using Orange workflows"}](/workflows/workflows.pdf)