This week #workoutwednesday challenge was set up by Ann Jackson. It was slightly difficult challenge as this week it was about finding signals or patterns in data with use of control chart. It was more about finding statistical signals using standard deviations and mean/ medians with signals to indicate change in pattern or trend or finding an outlier within the dataset. This is good way to analyze the data and get statistical insights/ signals from the data about pattern and behavior.


Available here

Data for the workout can be downloaded from here

Here is my output for the challenge (Click on Image for interactive version):

It took me time to understand the requirement and figure out the way to design the visualisation. lookup() & windows functions did help a lot to build this requirement

Let me share the steps required to create this visualisation post data is imported into Tableau:

Step 1: Create 2 parameters for the requirement

Select a Middle line: This parameter to provide drop down option for either median or mean

Select A Test: This parameter to select the type of test we want to perform (outliers, Trend or change)

Step 2: Create the calculated fields required for the visualisation

Middle Line: This field is based on parameter “SELECT A MIDDLE LINE” and based on selection we will have either median or mean of sales


+3SD: This field is to calculate + 3 standard deviation based on middle line

-3SD: This field is to calculate – 3 standard deviation based on middle line

TEST – Outliers: Boolean field to highlight the outliers in the dataset i.e. if the sales value is either above +3SD or -3SD


TEST – TREND: Boolean field to highlight the trend whether current sales > previous month > previous to previous month or current sales < previous month < previous to previous month.

TEST – CHANGE: Boolean field to highlight whether 3 consecutive fields are below or above middle line.

TEST – SELECTION: This field is based on “SELECT A TEST” Parameter and uses above 3 calculated field to show the required value.

Tooltip – Signal: Tooltip to show signal if the sales value matches with any of the parameter selection criteria

Show Text: This is the field to show the necessary text based on parameter selection

STEP 3: There are 3-4 more fields which we need to create but lets start creating data visualisation for first 2 sheets

Drag Order date to column shelf (convert to month)

Drag Sum(sales) and TEST Selection to rows shelf then change it to dual axis followed by sync axis

Drag Middle line, +3SD, -3SD to details in Marks Area

Add a 3 reference lines on the basis of Middle line, +3SD and -3SD

Add Parameter control for Parameters “Select a Middle line” and “Select A TEST”

Then format the sheet as per requirement with tooltips and it will look like following:

Create an new sheet for Text data and drag Show text field to Text

Step 4: Final sheet for Monthly strip chart

For Strip chart, we will need to add following fields into the dataset

TREND: This is boolean field which we will use to highlight the pattern for Trend

CHANGE:This is boolean field which we will use to highlight the pattern for Change

SymbolCOLOR: Based on Parameter selected for the test, this is will return values for the test select and we will use this as also to segregate data into 3 color bucket of blue, amber and orange to show following:

  • Meets the test criteria
  • Part of test pattern

Finally, we will create 3rd sheet with strip chart:

Drag Order date to column shelf (convert to month)

Then Drag symbolColor to color and shape area under Marks area then create the tooltip to show the values

Step 5: Create the Dashboard

Add all 3 sheets into dashboard as per below image:

Now we are ready with visualization with user experience which we wanted to show.

Click here for Tableau file

Thanks Ann Jackson for this workout.

Happy Data Visualisation!!!!

Thanks for visiting this post. Please do let me know your feedback or if you have any questions about the blog do not hesitate to contact me on twitter (@Desaimithun)

Do subscribe to to keep receive regular updates.

In my previous post, we learned about text mining and sentiment analysis on News headlines using web scraping and R. Text analytics has been one of the black boxes of analytics. In this post, we will dive into text analysis of headlines with simple Natural Language Processing(NLP) using UDPipe in R. UDPipe provides language-agnostic tokenization, tagging, lemmatization and dependency parsing of raw text, which is an essential part in natural language processing.

Step 1: Dataset
We will use the same dataset which we created in last article from ABC News. We will take all the headlines which have been published on ABC news for the year 2018.



abc_scrap_all <- readRDS(file = paste0("abc_scrap_all.rds"))
#Remove duplicates
news <-  abc_scrap_all[!duplicated(abc_scrap_all$headlines),]

news %>% group_by(Date) %>% count() %>% arrange(desc(n))
## # A tibble: 121 x 2
## # Groups:   Date [121]
##    Date         n
##    <chr>    <int>
##  1 20180523   213
##  2 20180504   172
##  3 20180320   155
##  4 20180220   153
##  5 20180227   152
##  6 20180301   148
##  7 20180423   147
##  8 20180208   145
##  9 20180321   144
## 10 20180322   137
## # ... with 111 more rows

Step 2: pre-trained UDPipe model
UDPipe package comes with pre trained model for more than 50 spoken languages. We can download the model using udpipe_download_model() function

#model <- udpipe_download_model(language = "english")
udmodel_english <- udpipe_load_model(file = 'english-ud-2.0-170801.udpipe')

Step 3: Annotate the input text
use udpipe_annotate() to start with udpipe and this will annotates the given data and put the variable into data frame format with data.frame()

# use udpipe_annotate() for analysis 
textanalysis <- udpipe_annotate(udmodel_english, news$headlines)
#data frame for the output
textframe <- data.frame(textanalysis)

Step 4: Universal POS (Part of Speech)
We will plot Part of speech tags for the given headlines

## POS
POS <- txt_freq(textframe$upos)
POS$key <- factor(POS$key, levels = rev(POS$key))
barchart(key ~ freq, data = POS, col = "yellow", 
         main = "UPOS (Universal Parts of Speech)\n frequency of occurrence", 
         xlab = "Freq")

plot of chunk unnamed-chunk-3

Step 5: Frequently used Nouns in headlines
Lets plot the most frequently used nouns in headlines

noun <- subset(textframe, upos %in% c("NOUN")) 
noun <- txt_freq(noun$token)
noun$key <- factor(noun$key, levels = rev(noun$key))
barchart(key ~ freq, data = head(noun, 20), col = "cadetblue", 
         main = "Frequently used nouns", xlab = "Freq")

plot of chunk unnamed-chunk-4
More than half of the top nouns used in headlines seem to be indicating negative atmosphere

Step 6: Frequently used Adjective in headlines
Let’s analyze the Adjective used in headlines as its a news website which will love to magnify and inflate using several adjectives

adj <- subset(textframe, upos %in% c("ADJ")) 
adj <- txt_freq(adj$token)
adj$key <- factor(adj$key, levels = rev(adj$key))
barchart(key ~ freq, data = head(adj, 20), col = "purple", 
         main = "Frequently used Adjectives", xlab = "Freq")

plot of chunk unnamed-chunk-5

Step 7: Frequently used Verb in headlines
Do headlines bring in any sign of optimism or just infuse pessimism? The kind of Verb used by media house can certainly help in highlighting direction of optimism or pessimism.

verbs <- subset(textframe, upos %in% c("VERB")) 
verbs <- txt_freq(verbs$token)
verbs$key <- factor(verbs$key, levels = rev(verbs$key))
barchart(key ~ freq, data = head(verbs, 20), col = "gold", 
         main = "Most occurring Verbs", xlab = "Freq")

plot of chunk unnamed-chunk-6

With words like dies, killed, charged, accused and many more it does not look like ABC news is not interested in building an optimistic mindset amongst its citizen. It is just acting like a media house which will look into hot, sensational or burning news to gain further viewership.

Step 8: Automated keywords extraction using RAKE
Rapid Automatic Keyword Extraction(RAKE) algorithm is one of the most popular(unsupervised) algorithms for extracting keywords in Information retrieval. It looks for keywords by looking to a contiguous sequence of words which do not contain irrelevant words.

## Using RAKE
rake <- keywords_rake(x = textframe, term = "lemma", group = "doc_id", 
                       relevant = x$upos %in% c("NOUN", "ADJ"))
rake$key <- factor(rake$keyword, levels = rev(rake$keyword))
barchart(key ~ rake, data = head(subset(rake, freq > 3), 20), col = "red", 
         main = "Keywords identified by RAKE", 
         xlab = "Rake")

plot of chunk unnamed-chunk-7

Step 9: Phrases
Now, we will extract phrases in which a noun and a verb forming a phrase. Let us bring out the top phrases that are just keywords or topic for this headlines data.

## Using a sequence of POS tags (noun phrases / verb phrases)
textframe$phrase_tag <- as_phrasemachine(textframe$upos, type = "upos")
phrases <- keywords_phrases(x = textframe$phrase_tag, term = textframe$token, 
                          pattern = "(A|N)*N(P+D*(A|N)*N)*", 
                          is_regex = TRUE, detailed = FALSE)
phrases <- subset(phrases, ngram > 1 & freq > 3)
phrases$key <- factor(phrases$keyword, levels = rev(phrases$keyword))
barchart(key ~ freq, data = head(phrases, 20), col = "magenta", 
         main = "Keywords - simple noun phrases", xlab = "Frequency")

To conclude, we see here is commonwealth games and gold coast being top used phrases as Gold coast was hosting commonwealth games this year. Also, US influence on the news headlines with Wall street, white house and Donald trump being used frequently in headlines.

Hope this post helped you to get started with text analytics and NLP in R.

Please do let me know your feedback and if any particular topic you would like me to write on.

Do subscribe to to keep receive regular updates.

After writing previous article on Twitter Sentiment Analysis on #royalwedding, I thought why not do analysis on ABC news online website and see if we can uncover some interesting insights. This is some good practice to do some data scrapping, text mining and use few algorithms to practice.

Below is the step by step guide:

To start with we will scrap headlines from ABC news  for the duration of entire 2018. To get historical headlines, we will scrap abc news homepage via Wayback Machine. Post few data crunching and manipulation we will get the clean headlines.

Step 1:
Load all important libraries which we will be using in this practice

#Importing Libraries

#Path where files output will be stored

Step 2:

Use wayback API call with and pass this information into json with the text content.
Here we will also filter the time stamp to have dates from 1st Jan’18.


#API request
req <- httr::GET(AU_url, timeout(20))

#Get data
json <- httr::content(req, as = "text")

api_dat <- fromJSON(json)

#Get timestamps which will be used to pass in API

#Reverse order (so recent first)

#Scrap the each and every URL to get headlines from website

#Filter time_stamps to have dates after 2018

Step 3
We will create an loop where we will pass URL with necessary timestamp to get all headlines which were published from 1st to 21st May’18 on abc news website. Also, we will remove all duplicate headlines which we might have got while scraping the website.

#Dataframe to store output and loop to get headlines
abc_scrap_all <-NA
for(s in 1:length(time_stamps)){





   print('Valid URL')

    #Scrap the data from URLs



      Date <- substr(time_stamps[s],1,8)

      #Get headlines
      headlines<-initial  %>% html_text()

      comb<-data.frame(Date,headlines,stringsAsFactors = FALSE)

      #Remove NA headlines
      comb<-comb[!($headlines) | comb$headlines=="" | comb$headlines==" ") ,]

      #As a df

      #Remove duplicates on daily level
      comb <-  comb[!duplicated(comb$headlines),]


        #Save with the rest



#Remove duplicates
abc_scrap_all_final <-  abc_scrap_all[!duplicated(abc_scrap_all$headlines),]

Step 4:
We will use the headlines and do sentiment analysis on the headlines using Syuzhet package and see if we can make some conclusion

abc_scrap_all_final$headlines <- str_replace_all(abc_scrap_all_final$headlines,"[^[:graph:]]", " ")
Sentiment <-get_nrc_sentiment(abc_scrap_all_final$headlines)

td_Rowsum <- data.frame(rowSums(td[2:1781])) 

#Transformation and  cleaning
names(td_Rowsum)[1] <- "count"
td_Rowsum <- cbind("sentiment" = rownames(td_Rowsum), td_Rowsum)
rownames(td_Rowsum) <- NULL


qplot(sentiment, data=td_Plot, weight=count, geom="bar",fill=sentiment)+ggtitle("Abc News headlines sentiment analysis")

Conclusion on Sentiment Analysis:
Human brain tends to be more attentive to negative information. To grab attention of readers, most of the media houses focus on negative and fear related news. Thats what we see when we analyzed the abc news website headlines as well.

Lets analyze further on headlines:

Wordcloud for frequently used words in headlines

  corpus = Corpus(VectorSource(tolower(abc_scrap_all_final$headlines)))
  corpus = tm_map(corpus, removePunctuation)
  corpus = tm_map(corpus, removeWords, stopwords("english"))

  frequencies = DocumentTermMatrix(corpus)
  word_frequencies =

  words <- colnames(word_frequencies)
  freq <- colSums(word_frequencies)
  wordcloud(words, freq,
            min.freq=sort(freq, decreasing=TRUE)[[100]],
            colors=brewer.pal(10, "Paired"),

plot of chunk unnamed-chunk-5

Surprisingly, being an australian news agency most frequently used word in headlines related Donald trump (American President) followed by police, commonwealth games, sport and australia

Find word associations:

If you have any specific word which can useful for analysis and help us identify the highly correlate words with that term. If word always appears together then correlation=1.0 and in our example we will find correlated words with 30% correlation.

findAssocs(dtm, "tony", corlimit=0.3)
## $tony
##      abbott headbutting       cooke        30th     hansons  benneworth 
##        0.64        0.35        0.30        0.30        0.30        0.30 
##     mocking       astro        labe 
##        0.30        0.30        0.30
#0.3 means 30% correlation with word "tony" 

That’s all for now. In my next post we will look further into text analytics using udpipe and see if we can build more on text association and analytics

Please do let me know your feedback and if any particular topic you would like me to write on.

Do subscribe to to keep receive regular updates.

Today’s blog we will learn about running R codes with Tableau. R is the most used statistical tools by data scientests and researchers across the globe. Together, R & Tableau could be extremely useful in data science arena as it can help organisations to unravel end to end discovery needs.

Here in the blog we will go step by step process of integrating R & Tableau

Prerequisite: R studio & Tableau already installed on machine

Step 1

Install Rserve Package on your R software

install.packages("Rserve", repos="")
## Error in install.packages : Updating loaded packages

Once the package is installed, we will run below command to load the package

## Starting Rserve...
##  "C:\Users\HP-LAP~1\DOCUME~1\R\WIN-LI~1\3.4\Rserve\libs\x64\Rserve.exe"

This will start the server in the background to enable Tableau to execute commands on R

Step 2

Now we will connect Tableau to R server using following

Help –> Setting and Performance –> Manage external Service connection

This will open the below shown window with auto populated values for Server as “Localhost” and port as “6311”.

Click on Test connection to check the connection, it will show successfully connected to Rserve in pop-up

click on ok
This will help Tableau to execute script commands like:

Step 3

We will take below FMCG example as our data source and import into Tableau

Step 4

Now, we will create an calculated field “Outliers” using following code to find outliers in the FMCG data:

IF SCRIPT_REAL(“library(pracma); a <- rep(1, length(.arg1)); a[findpeaks(.arg1,threshold=quantile(.arg1,.99),sortstr=FALSE)[,2]]=0;a;”, SUM([Sales])) == 0
THEN “Outlier”

One more calculated field “Cluster” with following code

‘set.seed(42);result <- kmeans(data.frame(.arg1,.arg2,.arg3,.arg4), .arg5[1]);result$cluster;’,
max([Sales]), SUM([Profit]),min([Freight Expenses]),MIN([Discount offered]),3)

Step 5

Drag order date on column shelf and sales on row shelf then drag “Outliers” into color

As you can see all the outlier values are highlighted in orange while other values are in blue color. This way we can find outliers in our data very easily by using dream team of Tableau and R

Step 6

Lets go one step further and see if we can do clustering in Tableau with R.

Drag Sales into column shelf and profit into row shelf (unselect analysis –> aggregate measure) then drag calculated field “Cluster” to color in marks area and add trendline in the chart. We will see it creates 3 clusters as shown below:

Hope this blog helps to understand how to run Rserver in background and execute R scripts using Tableau.

you can download tableau file from here

Thanks for reading the blog

Do subscribe to my blog and keep receiving new posts 🙂

##R markdown to WordPress
In this post we will look into steps to publish R markdown directly into wordpress. Here are the steps:

#1 Packages Needed:
Following packages needs to be installed on R studio


#2 Once the packages are installed write the blog or r markdown document:
Once R markdown is completed then to upload the post on wordpress, first ensure RWordPress is loaded


Then connect with your wordpress site with credentials:

options(WordPressLogin = c(user = 'password'), # your user name & password
        WordPressURL = 'http://tabvizexplorer/xmlrpc.php') # your WP url + /xmlrpc.php at the end

#3 Finally use knitr library to create an html code to be uploaded on wordpress

knit2wp('WP_markdown.Rmd', title = 'How to Upload R Markdown Directly to WordPress', publish = FALSE) # your filename and blog post title

Done! your R markdown is ready to be publish